The 2sf Overhaul Project by unknownfile at 11:23 AM EST on January 9, 2009
Time to get started on some major revisions to the format. As I can't upload the changes at the moment, you get to read this here.
__ begin 2sf_spec_revision.txt __
2SF specification revision as of 01/09/2009 written by unknownfile, 2009 part of the 2sf overhaul project
This file was written to address some long-overdue changes to the 2SF specification. Sorry this came so late, but it's kind of hard to do when people go off and make a billion rips while you're still trying to sort some issues out with the format.
Changes in brief: a. Removed Desmume savestates from the specification altogether due to possible incompatibility issues. b. Defined "NEW2SF!" header in reserved section. c. Created a custom savestate format for legacy rips. d. Removed _frames tag as it is no longer needed.
Part A. Desmume savestates are no longer valid.
Savestates in the Desmume format are no longer valid due to possible revisions in Desmume rendering certain savestates invalid. This is also to leave room for possible support across multiple emulators.
Part B. NEW2SF!
All 2sf files ripped after this format revision must contain the following 8-byte string to differentiate them from old-format 2sfs.
'N', 'E', 'W', '2', 'S', 'F', '!', 0
Part C. The 2sf savestate format.
This is a custom savestate format used on rips that require them. The header goes something like: 4 bytes '2', 'S', 'A', 'V' 4 bytes unsigned int size
The savestate is stored sparsely using the following command:
4 bytes 'C', H', 'N', 'K' 1 byte address map to write this data to (0 = ARM9, 1 = ARM7) 4 bytes size xx bytes data
At the end of every savestate is a 3 byte string, "EOF". This should be checked when the save is initially loaded. If EOF is not at the end of a savestate it can be considered invalid.
Part D. Removed _frames tag.
This is a needless hack that was used in early rips so that they would work in Highly Whatever. Now that vio2sf can handle these rips without using the _frames tag, it is no longer supported.
Lunar Knights was missing a few tracks, and the streams. It could do with a re-rip.
Aside from that, I'd like to examine Castlevania: Order of Ecclesia to make sure everything's all right, but it apparently isn't being released or something, despite the fact that there's no CD commercially available for the game.
And be careful with the Phoenix Wright stuff. 3 was missing some ringtones and other jingles that were dismissed as sound effects before.
The two games I want most are Bust-A-Move DS and (as you've already heard over a million times -.-) Pokémon Diamond. Due to custom drivers and headers though, I know at this date they are "impossible" [possibly not, but time consuming if possible]. Plus the fact that the first is technically anime.
followed by the more notable games
Mario Kart DS? The current rip, is missing Final Lap tracks and tags.
Also, I am assuming there will be a new plugin for these changes.
Also,(2) does this mean you are breaking free of Desmume? Or still planning too?
And will eventually, in the far, far, far future, channel splitting be possile? Or does the format that is currently being built make this very difficult?
MKDS tags would be great. I had an Mp3 rip which i was gonna use, but it didn't have as many tracks as the DS version, so i didn't bother. I figured there'd be someone familiar with the game who could tag it.... guess not!?
"CHannel splitting" can't really be done on the DS. That is, unlike the NES (NSF) or SNES (SPC), which have specific hardware registers that can be enabled/disabled, the DS does most of its stuff through software. Yes, I'm aware that the DS does have hardware registers which can be enabled/disabled, but they're not used in the same way as the SNES/NES were. The "mixing" is done in software. Think of it this way - with a MIDI, you can turn instruments/channels on/off. But that's done using the program that's playing the MIDI file. You can't alter the output by disabling certain pieces of hardware on your computer (excluding muting the left/right stereo channels). 2SFs are similar in that the "player" doesn't actually play anything. It's an emulated environment that allows the NDS software (contained in the 2SF) to run. This software has been modified to output only audio (supposedly), which is why you get music from 2SFs. To alter the output, the only thing the "player" can do (which is an emulated environment) is to disable/enable pieces of hardware. Like your PC, this isn't going to do much to alter the sound output.
In other words, it's not a limitation of the 2SF format that's stopping you from muting channels, as much as it is the NDS hardware that's stopping you from doing it. However, it should be possible, with enough skill and expertise, to modify the software in the 2SF to play back specific channels (this would have to be during the ripping process. It's not something that can be implemented in the player for the reasons stated above), just as you can with USFs.
As for question 2 - my understanding is that UF is simply trying to make the 2SF format more compatible. For example, USFs are tied to Project 64 pretty tightly. I think it's because USFs use a savestate to bypass the N64 boot code. This savestate is in the PJ64 format. It's one of the reasons you don't see more USF players out there. At least, that's my understanding. If I'm not mistaken, UF is imply trying to avoid this problem. As far as I know, he has no plans to no longer use DesMume, but I haven't actually asked him.
Hopefully that answers your question(s). Mouser X over and out.
i can't be bothered to read all that, but the 2sf config seems to have channel muting (16 MIDI channels I am guessing, so it'd probably only work with generic driver games - i haven't tested it out.)
That's not that much... Besides, you can certainly skim it fast enough. As for muting channels in the way you're proposing, it would require that the player be able to modify memory dynamically, as it's running. And that would only work on "generic driver" games that happen to work exactly as the player expects them to (using the same memory and whatnot). This is very unlikely to happen (that games happen to do the exact same thing in the exact same piece of memory). With the current 2SF rips, it might be possible, because most of them are using the Yoshi's Island driver (and thus might use the exact same memory to do the exact same thing). But this is not the ideal situation, and when more rips are done "properly" (using their own driver), the method you seem to propose will almost certainly break.
I know of the 2SF config thing you're referring to, but I haven't experimented with it enough to find out what it's doing. Mouser X over and out.
What Mouser X is trying to say is that each channel is allocated dynamically (as per usual with MIDI sequencers). So each note of a lead section can span multiple hardware channels.
Mario Kart DS? The current rip, is missing Final Lap tracks and tags.
Aren't the Final Lap versions just the same song played a bit faster? Is that really necessary? And if you think an UF rerip will magically give you a tagged set, you're in for a nasty surprise.
I'd love to see FFTA2 rerip (since the current one has about 50 MB of unneeded data) and, dunno, Soma Bringer (though if Procyon Studio sequencer is as insane as their streams, don't think anyone sane is capable of ripping it) would be nice to have.
by unknownfile at 11:15 PM EST on January 10, 2009
@MouserX: Ahh, I was wondering why the same thing hadn't been implimented into 64th Note. Surely if VGMTrans can export as MIDI, there are set channels it uses, but as Knurek said they are dynamically allocated making the process nearly impossible.
@Knurek: Yes, the Final Lap versions are 100% necessary, lol. ;-) (They are in the Mario Kart Wii rip [Which is the only one that has them])
If you couldn't modify USFs to mute channels, then Banjo-Kazooie (and Tooie) and other games with dynamic tracks couldn't be ripped. On the other hand, maybe the ripper isn't muting channels (in software mind you, which the emulator can't effect) so much as they are deleting the samples used in those channels, which, I would think, would essentially have the same effect. If that's what's happening (disabling the samples somehow, but leaving the channel playing (resulting in silence)), then I would think that a similar method could be used in 2SF ripping. That is, removing samples from a particular song to, in effect, mute channels.
This method can't be used on the current 2SF ripping method (which uses the Yoshi's Island driver) because the audio data is being indirectly handled by the ripper (they never actually "see" the sequence/sample data in a form that allows them to modify it appropriately. I suppose you could attempt hex editing stuff, but that would require a fair amount of knowledge about the files you're working with. The auto-ripping method does not require this level of knowledge. It's almost as easy as "point and click" to rip 2SFs). To successfully rip dynamic 2SFs, I can see it being done in 1 of 2 ways.
1) The game is ripped properly, with its own original driver. During this process, the ripper identifies how the music is put together in memory, and realizes how to mute channels. With this knowledge, they rip the tracks as you'd expect to hear them in-game.
2) Someone who's familiar with the "generic driver" sequence format would need to hex edit the various files to, I would assume, remove certain pieces of the sequence, resulting in "muted" channels. These modified files could (theoretically?) then be inserted into the Yoshi's Island driver, and you'd basically end up with a "generic rip-kit" rip, using those modified files.
I don't know if method 2 would even work, but as far as I know, it sounds like it could. However, I wouldn't recommend that method, as it sounds very hackish to me (though, there are those who would say the same thing about the current 2SF ripping method).
I suspect that for the impatient/ADD people, I wrote to much. Sorry about that. In short, yes, you can do that with USFs. Mouser X over and out.
by unknownfile at 12:43 AM EST on January 11, 2009
Actually I do. The games that either have an OST or a soundtest and, you know, ones that I like. Did at least the Phoenix Wright games, Layton, Order of Ecclessia and Knights in the Nightmare, prolly more.
Mouser X: I wasn't proposing anything, I was saying it's already there. However, yes, voices/parts seem to be allocated to the channels dynamically, now that I've tested it. It's sometimes that way with SPCs too - I call it trackfucked :D and yes, deleting samples works quite well in those instances, i've done it with SPC2IT conversions while transcribing. Problem is when multiple channels/musical lines use the same sample.
Oh, I'm sorry; Lunar Knights is missing exactly one track, and exactly one stream.
077: Link Battle and 000: Dawn of the Sun.
And the Japanese pre-order bonus for Castlevania wasn't to be sold, only given to people that pre-ordered the game...in Japan. Thus not meeting the definition of 'commercially' available.
'Cause if I COULD buy it, I wouldn't be crying about the wait for the set.
*harumphs*
EDIT: Actually I do. The games that either have an OST or a soundtest and, you know, ones that I like. Did at least the Phoenix Wright games, Layton, Order of Ecclessia and Knights in the Nightmare, prolly more.
So you're the idiot who keeps timing fades to seven seconds? Man, do you ever make more work for me. Not to mention your Japanophilic tags. :P
I am assuming you are referring to Mouser, because I wouldn't be much help... Not to mention the fact I don't have an IRC client or know the channel.
The only times I use Hex are in Computing classes (converting from binary to hex and vice versa) and on Warcraft III, where I export abilities and use hex to overflow the values, because Blizzard don't allow -ve integers. =/ So I manually do it instead, why not allow -ve integers?? They allow negative reals. Its just madness. (Sparta)
They are indeed, Study Leave for me starts on Tuesday...
by nensondubois at 6:04 PM EST on January 11, 2009
No, I meant I can't help with the 2SF format at all. I said either because I didn't have access to an IRC client as a reply to Elven's comment because he didn't.
edited 7:05 PM EST January 11, 2009
spec overhaul 2, coming soon by unknownfile at 9:54 AM EST on January 12, 2009
Next specification revision is in the works and will be released after 10 rerips are done in a project known only as The Savestate-free Challenge. If these rips are completed without a single savestate, guess what gets axed from the specs.
Made some more efforts to look at pwright again yesterday... here's what I found out:
- the sound select routine is called by the main thread - several other threads need an opportunity to run before a swi-trap is inserted otherwise the thing hangs, possibly due to interrupts being disabled
Lemme drop some science on you regarding the standard Nintendo driver.
* The actual guts of the driver are almost entirely on the ARM7 and use vblank for timing.
* The library on the ARM9 side mostly just queues up commands for the ARM7 part, so when a game calls e.g. "start sequence" the code it calls doesn't really cause the command to happen.
So it should be possible to create self-booting rips that require only the ARM7 and sound chip to play back (plus some fakery of the ARM9 end of the command FIFO, similar to what Corlett does for some PSF2s).
hcs, your right lol. They are STRM and there are 17...
by unknownfile at 11:35 AM EST on January 13, 2009
Thanks! I'll look into it when I get the chance.
edited 11:44 AM EST January 13, 2009
re: Phoenix Wright by unknownfile at 4:15 PM EST on January 13, 2009
Ripped.
next up by unknownfile at 11:34 AM EST on January 14, 2009
Meteos will be attempted tonight.
Three drivers this time around:
- sequencer - ADX player - stream player
initial analysis complete by unknownfile at 8:40 AM EST on January 15, 2009
The sequencer is easy as hell to find in this one.
The streaming driver is called up from a routine that's loaded into memory, but that's only for the title theme. It looks like it has to wait for a sequence to stop playing before a stream plays.
The ADX driver is hidden away in the overlay files and I still don't know how those things work, mainly because Google has failed me. Ah well.
Actually, disregard me not knowing about overlay stuff... it's actually data loaded into RAM at a specific address and can contain stuff like code etc. Source
edited 9:20 AM EST January 15, 2009
meteos brief update by unknownfile at 8:49 AM EST on January 16, 2009
Made progress on the ADX driver last night, still have a bit left to go before it's done.
and it's done by unknownfile at 3:19 PM EST on January 17, 2009
God dammit, I hate that ADX driver. Anyways, that's the first part of this rip done.
Worked a bit more on Meteos and an unnamed game over the weekend... Although I have exams all this week and next week so I still can't get anything done :|
HLE = what's supposed to be emulated by some sort of device (CPU, GPU, DSP etc) ends up being handled by routines written by the author of the emulator. Sorta hard to explain it, but it's a hacky way of getting stuff to work.
Going to be cleaning up Phoenix Wright for releasing tonight. I don't think it's going to be able to sit around for long, and I've decided that it's just better to release the things instead of delaying them to appease my own interests.
By the way, the problem with most existing sets is that savestates are unoptimized because no such support was included in the tracer. I never added an optimizer because I was lazy. Most sets could easily be fixed by optimizing the savestates, but it's just better to remove them entirely.
Still not much progress on Pokemon. No matter what I do, it always ends up going off into UNIMPLEMENTED OPCODE: 0x018203883 0x2800000. Bleh.
I think it's time to put this aside and focus on other games for now.
Super Princess Peach by unknownfile at 3:06 PM EST on January 28, 2009
Ripped. Still got the Nintendo logo routines running so I'll probably fix that later.
you knew this was coming by unknownfile at 10:32 PM EST on February 5, 2009
dumpan liek spcs by unknownfile at 11:06 AM EST on February 11, 2009
R. Belmont said it is possible to do rips without the ARM9 running anything...
As much as I dislike SPCs, it is pretty much possible to dump stuff in this way, but I haven't been taking a look at the ARM7 code (which is loaded into shared memory then executed).
I don't like not having the best possible SPC player available. (Plus, it probably will never support RSNs or metadata.) All I can do is endlessly check this thread for a miracle update.
By the way, don't spend too much time there. The people suck.
by unknownfile at 10:21 PM EST on February 11, 2009