0. To make sure you "own" the game, you'll find yourself with three situations: 0.1. If it's a physical copy, find a way to dump that copy through your computer. After you've done that, go to the next step. 0.2. If it's a dumped copy(.iso), use exiso. You can find it here. 0.2.1. From there on, you'll have to extract it to some folder. It must not have special characters, only letters. 0.2.2. Start the program by using command prompt. If you're confused, just Shift+Right Click on the folder you just extracted and click on something that says "command line" or "command prompt". 0.2.3. The parameter for extracting the XBOX 360 .iso file with exiso is this: "exiso.exe -x [directory of XBOX 360 .iso file]" It will automatically create a folder. 0.3. If the copy is already extracted, with directories and all, then you can proceed safely to the next step. 1. Use quickBMS. You can find a download link for the latest version(0.7.6a). Repeat step 0.2.1. 2. Save this script on the extracted folder. 3. Repeat step 0.2.2. 4. Type this while on command line/command prompt: "quickbms.exe [directory of metal_gear_solid_xbarc.bms]* [directory of .xbarc file]* [output folder]**" or "quickbms_4gb_files.exe [directory of metal_gear_solid_xbarc.bms]* [directory of .xbarc file]* [output folder]**" * It is possible to drag-and-drop the file into command line/command prompt. ** In case you defined a non-existent directory folder, you will be asked by quickBMS if you want to create a folder or not. Just type "y", press enter, and you're done. Otherwise, if you defined an existent directory folder, then quickBMS will proceed extracting into that folder. 5. After extracting, there will be mostly .cl5 files on the folder you specified. Repeat step 0.2.2. then type "ren *.cl5 *.sdt". You'll see that the file extension of these files had changed.
Done! 1) Extracted the NTSC ISO of MGSHD. 2) I want to try MGS2, so I extracted all xbarc files using the QuickBMS script. 3) mgs2_1.xbarc is the only one that had cl5 files. 4) I renamed all cl5 files to sdt.
Funnily enough, here's another case (working on ZOE right now) of MPEG partitioned data with Sony ADPCM audio. Even the custom audio headers are the same as in ZOE.
However, I'm not sure how one would go about demuxing this particular MPEG structure, because it seems to start with non-audio/video headers (0x01B3 and 0x01B5 for the most part). The video stream is playable in MPC-HC at least.
EDIT: Oh, and I only looked at the PS3 file, so I can only speak for that version.
@Nisto - Just in time, eh? I'm uploading these two files that gave the tool these errors: E:\MGS_HDC>sdt_demux.py E:\MGS_HDC\PS3\BLUS30847\PS3_GAME\USRDIR\MGS2\mgs2\us\movievr\_bp\_ps3\ending.sdt 0x0000000C: unrecognized stream ID: 30001
Well, looking at the new files, I'm satisfied to see that I seem to have gotten things right from the first file; it's just that there are more IDs which needs to be mapped to filename extensions. I can probably just extract anything currently unknown to *.bin, but if you require proper extensions, I'll probably need all .sdt files that you can find.
So I've updated the script, and at least for now it extracts further unknown stuff to .bin. The last part before the extension is the ID in hex. If you can put up with a bit of "hexploring", you might be able to determine appropriate extensions yourself, by looking at the first few bytes of the files. All you need to do is add new mappings near the top of the .py file (can't miss it). If you do, maybe share an update for other people that may be interested?
by AnonRunzes at 12:52 AM EST on November 12, 2016
@Nisto - I'll do my best!
EDIT: It seems that some of the files such as p004_01_p01.sdt and p070_01_p01.sdt handles a different audio codec. The MEGA folder will be updated soon with a new file(p004_p070_ps3.rar and t01-02a1-2d_ps3.rar).
EDIT2:Here is an "updated" version of sdt_demux.py. Not only does it contain new formats, it also supports formats from the PS2/XBOX versions of MGS2, and from there on, ZOE2 The 2nd Runner and MGS3. Not all of it has been covered though, but this should be enough by now.
Question, do .pac files all have "PACB" as the magic (first 4 bytes)? In that case, I guess 0x04 should map to .pac as well. Anyway, thanks for sharing the update.
I had a look at the files with the different coding and I'm stumped myself. I tested wrapping it in an MTAF file, and all possible codings available through GENH, but I can't get it to play properly. Sorry. At least it seems like it should be possible to determine if the coding is PSX ADPCM, so that's good. I believe the byte at offset 0x0A in Konami's custom 16-byte header indicates that (0=PSX ADPCM).
@Koto - Well, it might be possible but considering that most of the in-game music for some games is in .sdx format... only time will tell when the .sdx files will finally be reverse-engineered.
@Nisto - "Question, do .pac files all have "PACB" as the magic (first 4 bytes)? In that case, I guess 0x04 should map to .pac as well." No. The .pac files are entirely different formats compared to .pacb. Just to give you a hint, these .pac files are one of those MPEG2 video formats, and it doesn't have "PACB" as the magic.
"I believe the byte at offset 0x0A in Konami's custom 16-byte header indicates that (0=PSX ADPCM)." Well, regarding that, I think it's best to just leave it to the vgmstream staff at this point.
edited 10:48 AM EST November 12, 2016
by AnonRunzes at 12:05 PM EST on November 12, 2016
These folders contain the .sdt files I extracted using my quickBMS script called mgs2_dat.bms which can be found here. There are two audio formats used in 0110.sdt's case, but only one of them can be converted into GENH... apparently.
EDIT: Added one file in the folder that triggered this error: E:\MGS_HDC>sdt_demux.py E:\MGS3\SUBSISTENCE\DISC1_SUBSISTENCE\MGS\VOX\0001.sdt 0x00000030: unregistered stream / unknown header ID: 0000000F
Oh, I forgot to ask something: is there any way to do the entire batch based on a .py file, in which it processes all files at once?
EDIT2:Here's an update od sdt_demux.py. So far the only changes for this is the addition of a new container, albeit with two new types.
"I believe the byte at offset 0x0A in Konami's custom 16-byte header indicates that (0=PSX ADPCM)." --- "Well, regarding that, I think it's best to just leave it to the vgmstream staff at this point." --- Unfortunately, I don't think there's much they can do in this case, primarily because of the fact that the developers would most likely need more to go on, in order to properly detect this format. There is no magic number in the header, and not even any known extension - it's just a 16-byte header with the bare minimum metadata.
Also, I'm feeling slightly stupid having written this script, because hcs already wrote something quite similar to my script for MGS3/4 which appears to work for the HD files you've uploaded so far as well. https://www.hcs64.com/files/demux_dat_03.zip
edited 10:17 PM EST November 12, 2016
by AnonRunzes at 11:23 PM EST on November 12, 2016
Well, I think that tool works too.
by AnonRunzes at 10:22 PM EST on December 12, 2016
Turns out demux_dat only extracts the audio files, but not the rest of the files stored within these .sdt files. Which is a stark contrast to sdt_demux.py in which it extracts everything based on the ID of each .sdt file as indicated by the header.
Oh, and may I ask you(Nisto) to have your script extract the audio files as-is(with the .sdx extension, of course) instead of being converted to GENH? Or at least implement something(like sh3-sdex in which it has two choices: to extract or build the sd.bin file) in which the user [that executes the .py file] decides if the 0x00000001 ID file should be extracted as .genh or .sdx.
------------------------------------------------
Anyway, I just wrote a quickbBMS script for handling .msf(PS3) and .9tav(PSVITA) files. I don't know how to handle the .xwma(XBOX360) files though since the dpds chunk size depends on the duration of the file it was encoded on. Here they are: mgs_hdc_9tav.bms - it requires the func_header_AT9.bms script to work mgs_hdc_ps3_msf.bms
There might be a few issues with these scripts though, so I might need some feedback if I ever get to work with these issues.
Okay, so here's a rundown of audio codecs(based on the 0x00000001 format) used in MGS2(referenced in offset 0xA): 0x00 - PlayStation ADPCM 0x01 - XOR(0x7)'d IMA ADPCM 0x02 - Unknown ADPCM type 0x11 - IMA ADPCM
There is a sample at that MEGA folder I posted that says "MGS2_MOVIE_0110.rar". The .sdt file has the 0x00010001 and 0x00020001 IDs extracted as .bin files. So I had to modify the sdt_demux.py once again to assign an extention to these ID formats(.sdx_1 and .sdx_2 respectively, to avoid confusion). Here.