The format has yet to be released; in the meantime, go use VGMTrans until I sort this crap out.
FORMAT SPECIFICATION:
Version byte 0x91 (might be subject to change)
Reserved section - SR64-format Desmume uncompressed savestate (dumped by Desmume 2SF)
Code section - zlib-compressed ROM
Reserved section MUST start with either "CODE" or "PATCH" otherwise it is consider invalid.
CODE specifies a savestate (which is immediately followed by an SR64 header, then the Desmume savestate header). PATCH should be only used in mini2sfs. The syntax is:
5 bytes: ASCII "PATCH"
Next byte: 01 (register) or 00 (memory offset)
If byte is 01:
Next byte: Register (r0 is 0, r1 is 1, etc)
Next 4 bytes: signed u32 data
If byte is 00:
Next 4 bytes: signed u32 offset
Next 4 bytes: signed u32 data
Further progress on the format, player, or rips will be posted here.
Might I suggest ditching the CODE/PATCH labels and just using something binary? Better yet, since CODE always starts with SR64, you could just look for the SR64, otherwise consider whatever is there a patch.
And, if I may give you A WARNING FROM BEYOND THE GRAVE: don't sparsify early, 0s may be more important than they seem...
ROM -> savestate -> patches
mini2sf patches in the code section rather than the reserved section will use the same "PATCH" format as seen aboved, only zlib-compressed. The savestate will not require PATCH/CODE stuff.
I'll also start on the tracer tonight, though it will dump uncompressed data to be saved later.
Also, it's a long weekend, so I should also get started on Highly Nitric (going with the Highly * naming convention...)
I can't wait to get Mega Man ZX on 2SF.
Because the Sawaru Made in Wario test you posted was quite big. 23MB for the savestate, 8MB for the zipped savestate.
EDIT:
I have a small tracer running now. It's very buggy, and I'm not even sure if it works correctly.
Here's the source to the tracer (add this into MMU_read32):
// very buggy tracer for 2sf ripping
writeoffset=adr-0x2000000;
if (writeoffset < 0) {
// data out of bounds, probably bios calls or similar
return T1ReadLong(MMU.MMU_MEM[proc][(adr >> 20) & 0xFF], adr & MMU.MMU_MASK[proc][(adr >> 20) & 0xFF]);
} else if (writeoffset>0x2000000) {
// assume this is out of bounds, too
return T1ReadLong(MMU.MMU_MEM[proc][(adr >> 20) & 0xFF], adr & MMU.MMU_MASK[proc][(adr >> 20) & 0xFF]);
} else {
u32 crap=T1ReadLong(MMU.MMU_MEM[proc][(adr >> 20) & 0xFF], adr & MMU.MMU_MASK[proc][(adr >> 20) & 0xFF]);
fseek(trace,writeoffset,SEEK_SET);
fwrite(&crap,1,sizeof(u32),trace);
return T1ReadLong(MMU.MMU_MEM[proc][(adr >> 20) & 0xFF], adr & MMU.MMU_MASK[proc][(adr >> 20) & 0xFF]);
}
marioman: I'm not really sure when the rips will be complete, though I am currently seeking technical help with the project (other rippers, etc).
edited 5:04 PM EST February 14, 2007
- improve tracer
- code *really early* public beta HN
- rip a game using the generic sequence player
Also, if anyone has the resources to compile libsnd, please step forward.
I'm going to need to get started on learning the Winamp API, or I can always rip off some data from 64th Note.
Because Desmume is GPL-licensed, I'll need to release the plugin and emulator code under the GPL license, otherwise a group of geeks will need to sue me because I violated their non-existant license.
The tracer I have written is absolute crap, and rather only reads from memory than ROM. I'm going to need to fix that but for now I can only just sit here and stare.
What's new:
- (very unstable and incomplete) tracing
What's still needed to be implemented:
- built-in 2sf generation
- 2sf "playback"
You can find it here
Now let me go to sleep, damn you.
As long as they're optimised correctly.
As it's based on Desmume, which I know supports them. No games using them can be played, but you can get to the Mario vs DK2 title screen, which plays a STRM format song.
However, I don't have the thing organized correctly, and I'm not willing to work on it tonight.
Also, here's another obstacle:
The ARM9 CPU and ARM7 CPU use two different memory banks.
Also, once the ARM9 and ARM7 executables load, they aren't going to be reading from ROM space for code anymore.
I am seriously considering the following layout:
code: zlib compressed ROM
reserved: sections with identifiers "ARM9" and "ARM7"
This will mean that I will need to completely rewrite savestate support for Desmume to work in the new environment.
I can't promise when I'll have the player (let alone the format) ready, but I'm expecting that it'll be released in May at the latest.
You may ask more questions as I keep working.
~P
Still haven't implemented loading support yet...
The savestates still don't contain ARM9/ARM7 executables, THOUGH THEY DO CONTAIN the other memory banks.
Also, when I have the Wario crap working in Winamp, you have an option as what I should rip next:
- Kirby Canvas Curse
- Yoshi Touch and Go
Kirby should be easier to rip due to the nature of its sound select.
In the meantime, stuff.
The New Super Mario Bros has some good music too.
edited 1:21 PM EST February 20, 2007
HN is the player (Highly Nitric).
HN = The player, Highly Nitric, as Jouw has already said.
Just clarifying the 2 in '2SF' more. Mouser X out.
Actually, none of the executable tracers work, to my knowledge. Since my modchip equipment will be coming soon, I will either be playing a lotta Gamecube backups (arrrr :) ) or using my bricked Gamecube as a soccer ball ( >( ).
Also cocks.
- Load clean ROM.
- Pause emulation.
- Load savestate.
- Load MINI2SF data (to be written about later).
- Load ARM9 code.
- Load ARM7 code.
- Set registers, etc.
- Run.
More Desmume limitations:
- Desmume doesn't emulate the DS's PSG.
- Desmume runs on an interpreter rather than a recompiler, and said interpreter isn't optimized (it is coded in C, whereas other interpreters usually use assembly code, which is MUCH faster).
The good news is that Desmume outputs 16-bit PCM in its WAVs, so I could reroute the WAV output to Winamp for Highly Nitric.
Last but not least, someone get off your ass and make a logo. I suck with photoshop/fireworks/etc.
Reasons:
- School
- Lack of C knowledge (which explains why Desmume 2sf is so unstable)
- Frustrating generic driver (song IDs are better than song offsets, dammit)
- Lack of help on the project
I'm going to stop work on the project if I can't get this thing working within April 1st (which is my deadline). I will continue work once I have a playable set and working player, even if it's so simple as a modified Desmume.
Until then, spread the word!
Since the tracer refuses data reads from the CPU programs, the following method of storage will be used for going about this nonsense:
- The NDS header will be read by a program whose job it is to clean out unneeded data in the ROM.
- The cleaner will set a "no-fly-zone" for certain data ranges (the header, ARM9 executable, etc).
- The data will be compared to a cleaned ROM file that has been traced from Desmume 2sf.
- Any data outside of the no-fly-zone and used clean ROM space will be erased.
Also, I am considering to do things differently with writing a player for this rubbish. I am going to release it as a Win32 commandline program rather than a Winamp plugin, as I find DLL programming too stupid and annoying. The source will be released, so anyone wanting to port the thing over to Winamp can do so. I don't want to bother with custom APIs with this much programming experience.
I'm not really going to bother with ROM space and optimization for this one, rather, I will just have the game patch stuff when it hits a certain offset, which works quite nicely. The sequence and sample data will already have loaded by then, so that's awesomeness on a stick. I'm not sure that I'll be able to rip all of the tracks as I'm doing all of this by way of the sound test.
Stay tuned.
Video of Desmume 2sf in action
I have decided to go with Winamp after all, btw.
Version byte 0x45
Reserved section specification:
The reserved section will be gzip compressed.
A "mini" savestate is a sparsed file, which is created using saveclean (to be released). It will compare two savestates against each other, and the data found to be the same will be changed into null data. This data is then gzip compressed.
When the 2sflib is loaded, it will load the ROM as per usual, and will pause once the ARM9 and ARM7 EXE's are loaded. The savestate in the LIB is then loaded into memory but is not loaded. The loaded data is then patched with the sparse data. Bytes found to be zero will be skipped.
After this is done, boom, we have EMULATION!!@!@@1904712u0hfiaw
As for a player, I'll start with Desmume 2sf and will move outwards into Winamp from there.
Oh, by the way, RAM tracing is going to be removed from Desmume 2sf in the next build, since the tracer does not read data in the executables. So there!
Another feature I'm thinking of implementing is something similar to SPC dumping. When a command is initiated, the emulator quickly checks through RAM space, and initiates a breakpoint at what it suspects to be SND_StartSeq. When the breakpoint is hit, GX_VBlankIntr will be patched into an infinite loop, and a savestate will be made. So that will try and take the learning curve for ripping off of this, eh?
Gzip compression must be retained to individual files (for gzread()/gzwrite()/gzopen()), so I will instead have savestates be zlib-compressed.
4 bytes: char identifier = { 'S', 'A', 'V', 'E', }
4 bytes: u32 savestate_compsize
4 bytes: u32 savestate_crc32
I hate having to revise this over and over but I need an efficient way to go about doing things.
And I am considering SR64 for savestates to avoid wasting time with reading 20.3 MB of null bytes.
And here's a mockup logo:
edited 12:33 PM EST February 27, 2007
fread(uncompbuf,1,0x15F91FE,savestate);
savesize=sizeof(uncompbuf);
printf("compressing savestate\n");
if(compress2(reserved,&savesize,uncompbuf,savesize,9) !=Z_OK){
printf("zlib compress failed!\n");
return 1;
} else {
fwrite(reserved,1,savesize,tsffile);
printf("compressed savestate, attempting to compress ROM\n");
return 0;
}
Note that this version requires you to have a savestate, but I am not sure when I will have a stateless version ready. As of now the format requires a savestate to be loaded to save time with booting.
http://unknown.hcs64.com/2sf/mk2sf.cpp
Compile with whatever you like. Requires the zlib libraries to compile.
Usually, the ARM9 processor will instruct the ARM7 processor to run the sequencer. Communication between the two processors, y'know...
Anyways, expect Desmume 2sf to support actual rips sometime in the future, though it is bound to be very buggy and stupid, knowing how good I am at C...
- English essay (still incomplete)
- History essay (3 pages minimum)
- Flash presentation (30-60 seconds)
I will do most of the work on that this week, and a bit on the weekend, providing that I decide not to be lazy.
However, I will be implementing 2sf support sometime throughout this week and March Break.
Anyways, I managed to compress Kirby Canvas Curse's ROM from 64 MB to 800 kb. The savestates contain all of the sequence data that has already been loaded. On average, that data is 50 KB (with samples and all).
uncompress(&uncompressedcode,sizeof(uncompressedcode),code,codesize)
Thankfully zlib has good ways of going about safe error handling so this thing didn't blow up and die.
EDIT: 2sf.c on pastebin: blah
edited 6:31 PM EDT March 12, 2007
No, decompression and playback isn't implemented yet, it's just up here so you can see what's going on.
Then I woke up, sweating.
Tomorrow, I will make what will hopefully be the last revision to the format. It will no longer be PSF based as I am no longer willing to use zlib capabilities, and because I don't want the format to be a cheap hack like USF. It will keep some compatibilities as PSF, such as mini/lib files and [TAG]s, but it will require a whole new container if I don't want this thing to become any more of a hackjob than it is already.
by Peter S. Conway
The 2sf format is a PSF-compatible format (meaning that it retains functionality of a PSF, but does not use the PSF container itself. The beginning of a 2SF file is similar to that of a PSF:
3 bytes: "2SF" (similar to "PSF")
1 byte : revision (in case the format needs to be updated or fixed)
Now this is where things get different:
4 bytes: unsigned long ROMsize - size of SR64 ROM
4 bytes: unsigned long ROMoffset - location of SR64 ROM in file
4 bytes: unsigned long savesize - size of SR64 Desmume savestate
4 bytes: unsigned long saveoffset- location of SR64 savestate in file
x bytes: the actual data itself
To avoid using compression, the SR64 sparse file format (as seen in USF) is used. Check the USF documentation for information.
Savestates should be specified as they help cut down unneeded information in the file, and skips over a lot of code. The ROM section MUST ALWAYS BE PRESENT, otherwise the 2sf is rendered invalid.</i>
edited 10:14 AM EDT March 14, 2007
DeSMuME, now with 2SF support.
It's pretty unstable, so don't come whining to me if it crashes.
Oh, and you're probably wondering about a 2SF to play around with, just to see if this thing works. Well, there's also one for you to take a look at:
Kirby Canvas Curse - Training
Took me about 4 hours to program in total so enjoy!
BTW, parts of this aren't GPL'd. I need to seek hcs's permission before I can release the full source.
This version includes the psftag files from 64th Note, which I am trying to get to work. You'll see a blank dialogue box come up when loading a 2sf, that's just what the parser managed to find in the TAG section.
now with live svajklubb
Oh, and I forgot to mention that you can actually load NDS ROMs with this version.
edited 10:57 AM EDT March 15, 2007
You need to compile this with VC++2005.
Stuff to implement:
- Savestate loading in memory (requires 21.4 MB... yuck)
- The ability for the thing to compile in Dev-C++ on Windows and possibly on other compilers on other platforms
- mini2sf/2sflib support
hcs, I would like to request help with this...
The old savestates have ceased working, so I am going to see what I can do with my builds.
In the meantime, looks like I'm going to break my promise of this format being finished by the end of March.
http://unknown.hcs64.com/2sf/DeSmuME_release03.rar <-- latest beta
Also, in order for this to run the Kirby Canvas Curse track, you'll need about 192 MB RAM allocated (according to Taskman). 22 MB of that is the savestate which I can't manage to de-allocate since it is a static buffer and is not malloced.
Anyways, report buegs and such here.
Also, this build does not have any PSF-compatibility left. It may be re-instituted in the future.
edited 6:45 PM EDT March 24, 2007
http://unknown.hcs64.com/2sf/DeSmuME_superbeta01.rar
You can now configure how long it takes between when emulation begins and when the savestate is loaded.
This whole timer business is a result of Desmume sucking immensely with savestates, which some of you have already seen (ie, the 2sf dumps do not work). I'm unsure why this happens but in either case, savestate support is buggy all across the board.
So see what you can get out of this build while I go troll facebook.
The new specs:
3 bytes: "2SF" (similar to "PSF")
1 byte : revision (set to 0x2 due to format revisions)
4 bytes: unsigned long ROMsize - size of SR64 ROM
4 bytes: unsigned long ROMoffset - location of SR64 ROM in file
4 bytes: unsigned long savesize - size of SR64 Desmume savestate
4 bytes: unsigned long saveoffset- location of SR64 savestate in file
4 bytes: init address
x bytes: the actual data itself
When the program counter hits the init address, the savestate will load and we will have a big party. Woohoo!
Also coming is Gzip compression for your 2sf needs. :)
I might also do optimizations on KCC this weekend, so hold onto your hats.
edited 1:02 PM EDT March 27, 2007
I am unsure where the problem is, so until somebody gives me a legal serial key for VC++2005, I am without debugging abilities at school.
And I also think it is rediculous that only one computer in the world (mine) can run this thing without it screwing up.
In the meantime, I'm going to see what I can do with the chips that arrived from Newark yesterday.
But with the summer coming up, you never know what will happen ;)
edited 2:21 PM EDT June 8, 2007
edited 5:14 PM EDT June 8, 2007
lib2sf.exe
somewhat optimized Phoenix Wright mini2sf set
Now excuse me, I must get back to studying for a test which I will almost certainly fail.
The newer games have a nasty habit of loading part of the game code in after the main executable's been loaded, so this makes it hard to disassemble. I'll post a complete guide on ripping stuff.
Also, now that I've got Yoshi's Island DS figured out, I'll be trying different sound banks with it. The cleaned ROM that I produced out of it last night, however, didn't work in lib2sf...
The 2sf rip page is now up. There you can view the status of stuff.
I've sent off correspondance to Neill to reserve PSF version byte 0x24.
edited 10:47 AM EST January 14, 2008
From what I've seen, rougly 90% of the NDS games use that, and with music data being easily extractable... why not ease up your life a little?
However, the YIDS way of ripping generic driver games seems to work well. I'm actually going to use it for the next few generic driver rips I'll be doing.
Mega Man ZX
Mega Man ZX Advent
New Super Mario Bros.
Sonic Rush Adventure
Contra 4
Kirby Squeak Squad
I have listed them in order of preference, but if you can get any of them ripped I would appreciate it.
Seems strange considering that all the files are just in freakin' plain sight (extracting them from SDAT is a cinch).
The generic sequence driver varies from game to game. It's impossible to fully automate the process because:
1) The sound driver might try and verify the sequence block. If it doesn't match or is invalid, it will not play anything.
2) Some games keep their sequence player code in RAM, making it impossible to view when someone looks at the code in ROM.
All requests have been noted. Tonight I will continue on the YIDS driver, and hopefully it'll be working later tonight.
I'd love to see Final Fantasy III/IV and Phoenix Wright 3 rips.
Kirby Canvas Curse
Super Mario 64DS
Mario Kart DS
Yoshi Touch and Go
Hotel Dusk - Room 215
New Super Mario Bros
Diddy Kong Racing
Zelda Phantom Hourglass
ta-da.
Yoshi Touch and Go
Today's serving of lib2sf
I tested lib2sf on my computer at school, and it works. Just takes a bit of time to load, though.
... and for some reason, Kirby Canvas Curse isn't working. Let me check some stuff out first.
Turns out I'm an idiot and that I muted it. It's working, but I'm ripping Mario Kart DS first because it's just as awesome.
edited 6:22 PM EST January 14, 2008
As you might have noticed in Yoshi Touch and Go, the infamous "dynamic track" problem has once again appeared to bother us. But you know what? If anyone complains, fuck 'em, they can just die in a pit.
Anyway onto what I promised:
Kirby Canvas Curse
Now, I'm going to update Phoenix Wright to work with the new player and I'm done for the night.
And it's updated. Good night guys.
edited 10:23 PM EST January 14, 2008
Also, does your format account for games that used streamed music (Deep Labyrinth uses STRM, Doki Doki Majo Shinpan ADX)? Of course you can simply exrtract the files and play them either in in_cube or convert to MP3/whatever, but still, would be nice to have them supported as well.
Phoenix Wright composer: Akemi Kimura, Masakazu Sugimori, Atsushi Mori
Kirby Canvas Curse composer: Jun Ishikawa, Tadashi Ikegami
Mario Kart composer: Shinobu Tanaka
Yoshi Touch & Go composer: Asuka Ohta, Toru Minegishi, Kazumi Totaka
And thx for artist credits.
This time I'm ripping both the Japanese and American versions for the vocal tracks. Hoorah!
Fuck the Japanese version, its vocalist sounds like your stereotypical kawaii-lolicon-bitch asking to be raped.
I will agree, the US version was better (even had satanic messages) than the Japanese version, and this was when Nintendo had not fully raeped the series to death (see also WarioWare Twisted).
The US version will be ripped as a result of Prime Blue being truthful.
Everything is working fine besides that, although my complaint is that it loads a bit slow, which is typical for zlib-packed data.
Here is a beta build. A lot of stuff is needed to be fixed and whatever, so you will experience crashes once every so often.
edited 6:24 PM EST January 15, 2008
1. zlib1.dll must be in your main Winamp directory for it to work. The dll is available on the Web. (Maybe it should be included?)
2. It seems to be a little CPU intensive, but I am sure that it will get worked out.
Nice job UF.
EDIT: Didn't see that up above about the main directory, trying it now ...
EDIT 2: The error disappears, but Winamp just gives a "Winamp has stopped working" error now instead of playing.
edited 8:04 PM EST January 15, 2008
But anyways I have a new build which I will put up when this set is finished ripping. (hint: it's Mario)
edited 8:34 PM EST January 15, 2008
both Castlevania games, though Dawn of Sorrow if I had to pick one
Diddy Kong Racing
New Super Mario Bros.
Super Mario 64 DS
edited 9:32 PM EST January 15, 2008
New Super Mario Bros
The player's also been updated, it should use a bit less memory when playback starts (the old one didn't free up the memory used by the savestate). It's at the same url so scoll up.
And yes, I know that track nsmb-000b runs at full CPU for a few seconds, and I don't know why it is, so go ask your dad.
edited 9:23 PM EST January 15, 2008
nsmb-000d <3
End Credits FTW!
OK, one more set and I'm out for the night.
Super Mario 64 DS
Yes, there are dynamic tracks, and no, I will not split them.
edited 10:04 PM EST January 15, 2008
There was something similar going with Highly Advanced a while ago.
From HA changelog: * Performance Boost - (decode_pos_ms - (mod.outMod->GetWrittenTime()-mod.outMod->GetOutputTime())) is now only called twice a second, rather than about 100,000 times a second, at all times. CPU usage is no longer 95-100% on most systems, when using XM-Play. If you have a slower system. (less than 1GHz, you might notice this performance boost in winamp as well.)
The thing is, I've looked at Task Manager and the plugin takes only about 8% of total CPU, so there's something other going on here (Silence detection perchance, this has some problems in current HA build as well, after disabling that, no problems).
I've tried my Winamp install as well (2.91 IIRC), and no luck here too. Even with the required DLLs (zlib1, msvcrt80) the plugin closes Winamp silently when starting to play a track.
Halp?
A note about the Yoshi Touch & Go composers: Kazumi Totaka didn't write any music for that game, he was just "Music Director." So, I'd reckon it's best he was left out of credits when it comes to it. *shrug*
EDIT: phoenix wright uses around 30 - 40% CPU for me.
edited 3:58 PM EST January 16, 2008
I'll fix the XMplay thing later if possible.
Also to let you guys know, this plugin is designed assuming that it's being run with Winamp 5.51, so I'd say it's time to get out of 2.xx territory.
Phoenix Wright is running at 50% CPU here, whereas all the generic driver rips are at 88%. For some reason, I think that another thread is running on the ARM9 despite it being locked up.
Wow, the generic driver rips really DO suck. There's slowdown all over the place with the generic rips, but Phoenix Wright runs full speed. I'm going to need to hunt down that extra thread and fix all the sets released thus far. This hardly has to do with a lack of optimization; it happened during cleaning with the full ROM.
Problem solved. Turns out that killing the video thread was causing the problem. Now fixing all the sets made from the generic driver so far, and they'll be uploaded in a bit.
All the problem sets have been updated. Get the fixed ones kthxbye.
edited 4:54 PM EST January 16, 2008
In other news, Metroid Prime Hunters is up. Shit music from a shit game, but I should get it out of the way before people flood me with more requests than I have now.
Also, the other rips might be missing a track, due to my sucky coding skills, so get searching for missing tracks as I'm too lazy to check myself.
Megaman ZX
edited 6:06 PM EST January 16, 2008