lazyusf2 by kode54 at 8:27 PM EST on February 20, 2015
I attempted a thing. It doesn't work much yet, at all. And not due to the R4300i cores, which all seem to work. Well, at least x86_64 old recompiler works. No x86 32 bit, no NEW_DYNAREC modes.
Thanks for the off topic reply. That's a feature I don't implement, and still don't feel like touching yet. Why can't rippers just make use of the NSFE format?
Thank you so much for working on this. The USF player as of now still isn't perfect so it's great to see work on this being done. Some USF files simply refuse to play and other rips have problems like DK64's pitch being slightly lower than it should (though maybe that has more to do with the rip than the player?)
Kind of off topic: I don't know much about N64's audio, but is it possible to add different interpolation filter options like Game Emu Player has for SPC files? (Cubic, Sinc, etc.) I'm wondering if this might help the quality of 22khz rips, even if it's not necessarily accurate to what the N64 hardware does.
@Knurek; download "not sofatso" for winamp; load up your NSF file, right click on the NSF file and select "View file info" then select "convert to NSFE". Then it should be separated tracks in Foobar when you load up that resulting NSFE (I believe).
Big :( @ Off Topic: TL;DR version - Knurek knows of NSFe, but there's no good, easy to use, and cross-platform tools to support the NSFe tagging (and transfering tags from one NSFe to a different NSFe file).
Knurek knows better than most of the NSFe format. However, he has an extreme distaste for said format. The reasons (as best as I can remember) for his view are that he hasn't found an easy, cross-platform NSFe tag editor (aka, there's no psfpoint equivalent for NSFe). Secondly (closely related to the first problem), when a NSF is re-ripped or updated, it's difficult to update the tags in a NSFe. For the M3U files, if a set is updated, it's a simple fix (in a text editor no less) to update the tags. In fact, often, no change is necessary at all. Just put the new NSF in place of the old one, and the M3U will work, without change.
Personally, I prefer the NSFe format, as it works in Rockbox, and is self-contained. The M3U files are a separate file, and is just another file to clutter up a directory. I am well aware that Knurek doesn't look at it that way, but if a good tagging tool were made for NSFe files, I think it would take care of many of Knurek's concerns. However, I am not Knurek, and I don't want to speak for him. I realize this whole response is largely "speaking for him", but my intention was to explain that Knurek knows perfectly well of the NSFe format, and doesn't need instruction on how to use it.
Regarding kode64's "Why can't rippers just make use of the NSFE format?" - Most of the NSF collections I've seen over the years have been compiled by Knurek. There have been other collections, and they rarely use NSFe. However, I don't often see rippers tag their own rips. And since NSFe is only useful for tagging, if the ripper isn't going to tag it in the first place, then NSFe isn't very important to them. On top of that, there *is* a tool that can convert NSFe files to NSF+M3U. But there is no tool to convert NSF+M3U to NSFe. Someone (snakemeat or ugetab, if I recall correctly) tried to make one, but they couldn't come up with a reliable way to parse the M3U file.
Anyway, that's a lengthy explanation as to why NSFe isn't a more popular, or more widely distributed, format. Mouser X over and out.
by punk7890-2 at 12:35 PM EST on February 22, 2015
Is it possible in the future of this version of lazyUSF to have a resample option like the SPC player does? Would be extremely useful for Rare's games.
Sorry about that. I can add the M3U reading, but not really offer much else.
Game_Music_Emu does provide its own viable and flexible M3U parser, for that purpose. It could be used for reading in M3U tags. Unfortunately, it cannot be used for writing back out the resulting tags. The classes lifted from NotSo Fatso could be repurposed and made portable for that, though.
I have added a resampler to lazyusf2, it defaults to outputting 44100Hz, with no real improvement in quality, but definitely an improvement in convenience.
I use the SoX resampler plugin all across the board for resampling in foobar. It has good sounding algorithm; I suggest people use that.
Hopefully there'll be an option for (auto)/(original sample rate) in lazyusf2
edited 2:01 PM EST February 23, 2015
by punk7890-2 at 10:06 AM EST on February 23, 2015
Yeah I do know about that. I use it all the time for Rare's awful output rate of 22khz in their games. It does indeed do the job. Was hoping in the future something close to or on par with SNESAPU's resampling methods since from my understanding do indeed add quality. Even if not, it sounds wonderful.
There is a difference here. I cannot override the mixing employed by the games, since they calculate a fixed number of samples at a pre-determined sample rate. Overriding that would require significant work.
So you still get 16KHz or 22KHz or whatever, cubic resampled and filtered by the game code and/or the HLE RSP, then upsampled to 44100Hz using a windowed sinc algorithm.
I could make the resampling optional, but my intention is to make it portable, and simple to use. I won't be including the SoX resampler itself, due to complexity.
Kode54: Since it wasn't obvious (or even hinted...) from my previous response in this thread (again, sorry about the "hijacking", but I got the impression some explanation/background would help to clarify Knurek's concerns over M3U support), I wanted to say thank you. You do awesome work. Hopefully a new "core/engine" will alleviate some of the problems that USF rips have suffered from (largely the previously mentioned speed issues). And while I definitely praise your efforts (and time spent of course), I doubt I'll be able to take advantage of your improvements, as I still use Winamp.
Even so, giving you a big "THANK YOU" is completely well deserved. Good luck, and I look forward to seeing your great, and continued, work. Mouser X over and out.
Ah okay. I thought it'd be hard to implant. Either way, using an external resampler is still better then just plain 22KHz . Thanks for your work! Will keep an eye on this.
Resampling would only diminish the quality. The reason I use it is because my DAC doesn't support sample rates below 44.1khz. It would be nice to see interpolation features though like cubic or sinc.
Games already use cubic interpolation, and there will not be a sinc interpolation option, even for HLE audio, since it requires storing more data than games are designed to store.
What I mean is, doing sinc interpolation requires me to make assumptions about the use of the cubic interpolation functions. The existing functions do not take a steady stream of samples, but rather, a steady stream of decompressed samples. I would have to know which stream is being interpolated at what time, which I don't think is really possible. ALists will likely process through stages of decompressing sample data to temporary buffers, then cubic resampling that data to other temporary buffers, then mixing those buffers into the next output stage. I don't know if the samples are all decompressed to different buffers, or to the same buffers. It would require significant analysis of AList processing.
I would at this point, even if Excitebike 64 is still broken. The presence of a recompiler speeds up execution greatly for most sets. I would even recommend using the HLE audio for most sets as well, except for a few cases, such as some tracks from Jet Force Gemini that use unknown AList functions.
I'm still waiting on a skilled RSP disassembler to work out how the UNKNOWN functions of naudio_mp3 work. I suppose I could take a look at that one later. I think it needs opcode 0 for some IIR filtering, other than the already known IIR filtering function.
Not sure if its the usf set itself or what, but just thought I'd point out a few weird clicks occur in Extreme G still. Can be heard on Race 5 and Race 8 tracks. Also, you don't have to trouble yourself over that one Jet Force Gemini track.
No, but it would help the emulation cause if that were fixed for HLE RSP, because default installs of Mupen64plus only use HLE, not an actual RSP emulator.
Don't mean to be a bother, but from my understanding is this resampler in sinc mode? I personally think its much better when resampling 22Hz songs using Cubic resample option since sinc doesn't mirror the lower frequencies. Would it be possible to add Cubic at least?
There's also a small bug when unselecting the resample option as it doesn't let you.
You want it to add frequencies that aren't there in the original signal? Does anybody know what the N64 actually does when playing lower sample rates? I strongly doubt that the DAC had a built in cubic resampler.
To best describe what I mean, I think your Multi-resampler does just this when setting it to Cubic. I think Jabo's audio plug-in for Project 64 does something similar to what I'm talking about also. I think it would be more efficient to do it this way rather the switch between Multi-resampler at Cubic set at 44100Hz for lower frequencies (22Hz).
From my experience, its much nicer on the ears. Hopefully that makes sense as I'm not technically inclined in this matter.
Kind of off-topic: I got the latest build (3/2/2014) off kode54's homepage; and if you all take the time to listen, you'll notice the resampler audibly decreases the fidelity of the USF playback when compared back to back with the original sample rate. So, that should shut people up about resampling making it "better". :P
To be fair though, with the SoX resampler you can't even hear the difference; depends on the algorithm.
I figured out the temp issue. All sets should be using the _enablefifofull tag, and any that break as a result of that tag need to be fixed. For now, add it to some games like 1080 Snowboarding or Legend of Zelda Ocarina of Time or Majora's Mask without any problems, but other games, like Super Mario 64, Conker's Bad Fur Day, and Goldeneye 64 break with this behavior.
Note that this behavior is actually normal for the system, as is the behavior enabled by the _enablecompare flag. It's just that many rips would not stand for that behavior as they are.
EDIT: I forget the name of the console tool, but you could also do a binary edit or append to the tag on the .usflib. If there's no tag, start one with "[TAG]", that's all caps.
OK, seriously... what the fuck is wrong with this fucking planet? WHY DOES EVERY FUCKING THING NEVER WORK AS IT SHOULD?
All I'm trying to do is edit the length of a USF, using the basic "Right click the file in foobar2000 and click Tagging>Edit length!
All I want is to set "Frantic Factory - Research and Development" to play for exactly 3600 seconds (1 hour)! But for some STUPID FUCKING reason the time doesn't always update!
Why do I want it this long? BECAUSE FUCK GRANT KIRKHOPE AND HIS STUPID FUCKING MAKING HIS STUPID SFX NOT HAVE THE SAME LENGTH AS THE REST OF THE SONG!
Edit: Let me guess... you can't make it longer than 59:59?
Edit 2: Never mind. I can't be bothered. Tell soneek that Mohammed Emwazi wants to rape him.
The actual tune in question has a loop of 56 measures (looping from measures 3-59), and the sound effects loop from for 31 measures (1 - 32). Therefore, the time it takes for them to be re-aligned, as it were should be the LCM of those two numbers, should it not? This is 1736.
Each measure has 4 beats, which means a total of 6944 beats. The tempo of this song is 120 bpm. 6944/120 = 57 minutes and 52 seconds.
However.... not only does the actual tune and the sound effects, not line up at that point, the tune doesn't even loop at that point (it's randomly in the middle of the tune).
This is due to a timing issue. Add the _enablefifofull=- tag to the .usflib to save yourself some time. Note that this tag will likely cause glitches with any other player than lazyusf2.
So I also fixed Turok 1 and Turok 2, by defaulting to disabling TLB Miss exceptions when a TLB write occurs.
Are you seriously going to listen to it for an hour just so you can hear the music and sound effects line up again after both have individually repeated dozens of times?
dogman91x: Change the file extension from *.usflib" to "*.usf", load the USF into fb2k, edit/add tags, then change the file extension back to "*.usflib". Assuming the set was ripped properly, this should work just fine (honestly, to my understanding, it's difficult to rip a USF, or any xSF for that matter, in such a way that that wouldn't work. You'd have to *really* mess things up very badly for that to not work).
Another option would be to use the tagging tool "psfpoint" to tag the USFLIB file. There's a few different versions out there, but technically they're all capable of adding/editing xSF tags just fine. Hopefully one of these 2 methods works for you. Mouser X over and out.
The foobar2000 method will not work, since it does not allow editing tags which begin with an underline. They get stored in the file's info fields instead of meta.
so with psfpoint you just use the line "psfpoint -_enablefifofull=- *.usflib" ? did I do this correctly? http://i.imgur.com/Sk0Sif3.png
Also it seems some stuff broke with the latest lazyusf builds, for example: Unable to open item for playback (Emulator appears to be stuck!): "C:\USF\Donkey Kong 64 (1999)(Rare)(Nintendo)\047 - Frantic Factory Car Race.miniusf"
Not that it matters but the Star Fox 64 usf set on joshw website sounded slow to me compared to ingame music.
Slow USFs by Sephirothkefka at 8:36 PM EDT on March 8, 2015
I've been noticing this as well with my USF's of SM64, SF64, and OOT. Conker and Kirby are fine. It only seems to affect games with that overall EAD soundfont.
Found you the OOT set still slow after adding the enablefifofull tag to the usflib? It seems faster to me, maybe not enough compared with the original music?
So.... I've found the OOT set is much better. However.... this is regarding Dinosaur Hunter...
I'll just paste this... I don't need to explain it, I hope:
Decoding failure at 0:26.814 (Emulator appears to be stuck!): "G:\Desktop\Video Game Rips\Turok - Dinosaur Hunter [Jikku Senshi Turok] (1997-02-28)(Iguana)(Acclaim)\06 Ancient City.miniusf"
The one that failed on was the one that was linked in this topic, which included the underwater version. It seems to be OK in Winamp with 64th Note though.
Wait, nope, they just play some kind of drums and nothing but (ironically the rhythm is completely different <_<).
Oh and I do have the set from usf.joshw.info. At least I'm sure I do. Having said that, the rip hadn't actually been updated since October 23, 2005.
So it seems adding that new tag to make things play at normal speed kills some sets.
Decoding failure at 0:30.550 (Emulator appears to be stuck!): "D:\Music\Star Fox 64 [Lylat Wars] (1997-04-27)(Nintendo EAD)(Nintendo)\30 Versus.miniusf"
Decoding failure at 0:30.434 (Emulator appears to be stuck!): "D:\Music\Star Fox 64 [Lylat Wars] (1997-04-27)(Nintendo EAD)(Nintendo)\31 Versus (One by One).miniusf"
Going to assume this kills all the tracks. The tracks play fine, then all the sudden after about 10 seconds these appear. This doesn't seem to happen on the Zelda games though.
http://joshw.info/?page_id=6 using the old version (2.1) of LazyUSF fixes the pitch issue with the DK64 rip. Wonder what broke with the newer versions.
The pitch of the instruments are slightly lower than how it should be (compare to the official soundtrack and an actual N64). The 2.1 version of the plugin seems to not have this issue.
Thanks for pointing out that bug, I've fixed it. LazyUSF2 now detects ROM region from the Project64 save state. In this case, the Donkey Kong 64 USF set was ripped from the PAL version.
is it really necessary to pull the usf_state throughout the whole emulator? you probably want to have a library, but wouldn't it be more useful to let the mupen64plus-core unchanged (except for some usf-specific init, execution and close methods) and by that be able just to link against libmupen64plus and not losing upstream connection?
Yes, it is necessary to maintain a unique state throughout the entire emulator core. Otherwise, there will need to be complex state swapping mechanisms, and import pthreads or similar to lock the emulator to a single thread at a time.
So, no. Unless you want to point out that libmupen64plus now uses a state machine block that I can stuff into usf_state?
I'm noticing errors in couple tracks of Hybrid Heaven: Decoding failure at 0:00.000 (Unsupported format or corrupted file): "D:\Music\Hybrid Heaven [ut] (1999-08-05)(KCE Osaka)(Konami)\sparse19.miniusf" Decoding failure at 0:00.000 (Unsupported format or corrupted file): "D:\Music\Hybrid Heaven [ut] (1999-08-05)(KCE Osaka)(Konami)\sparse64.miniusf"
I don't know what you mean by: "Only using this library to perform coverage checking instead of Project64?" Not sure if that was aimed at me or not.
I noticed some popping in a few tracks from Extreme G also. Most notably Race 5. I don't know where the old lazyusf player went for foobar, but it never had these errors.
Too bad I'm a big fan of improving things that aren't in need of improvement. I don't personally care that a handful of rips are broken by a switch to a faster and more accurate emulator, and that these games will likely need to be re-ripped using a whole new set of tools that haven't been invented yet.
The source code is available if anyone wants to try to fix it somehow. Pull releases welcome.
Unfortunately those tools probably will never surface. It's a shame that usf ripping isn't a thing anymore. Does anyone have a guide on how to rip a usf? I know no assembly but I can always bookmark it for later and try it (maybe) years down the road.
The problem is that there isn't really a good way of doing this crap that's documented, certainly the way I was going about it was bad. What guides there are refer to old, broken versions of Project 64 and Visual Studio 6, and ripping techniques that don't use much understanding. The result is the kind of broken rips kode54 is tired of hearing about.
What would be best is to take the idea and do new rips with modern, accurate emulators. Or better yet, come up with better ideas.
I've seen a few mentions of a document by someone42, that may be in reference to his extensive PSF ripping guide, though.
edited 12:29 PM EDT May 12, 2016
added some old freeshell.org files I'd forgotten about
edited 12:43 PM EDT May 12, 2016
I had forgotten about Nielk1's rip of "Battlezone: Rise of the Black Dogs" from 2014, this was probably the last USF rip done, so it is still possible.
edited 12:55 PM EDT May 12, 2016
unknownfile had two files mentioned here and here, but they seem to be lost.
edited 1:16 PM EDT May 12, 2016
---
Final words:
I feel bad about how USF turned out. I thought that I could hack something together without understanding how N64 emulation really works, and I was almost right.
The specification relies on Project 64 save states intimately. It would have been better to keep things based on ROMs so that the "ground truth" for a working rip would just be whether it runs on an N64, and any accurate emulator would work. I even had the equipment to test on hardware, but for some reason I took another path.
kode54 and Josh W have done a lot of work to hack USF onto other emulators, but I think approaching it from scratch is the way forward. When I started 12 years ago, the only attempt at N64 music ripping was an unreleased Winamp plugin to play sequences, which just didn't work in general due to variation between games. I think someone with determination and an eye for quality can make another step forward.
Well then, let's develop a new format for playing N64 music. This time, it'll be like this: 64sf - for sequenced music 64str - for streamed music
But instead of using emulators and all that hax, why not hack some N64 ROM to use it as a driver? Or maybe something that can detect all the sounds/music from a N64 game? But first we'll need to carefully study the N64 hardware. THEN it's where you need to start from scratch. What do you think PSF originated from?
Thanks for missing the point entirely. You get a cookie.
Plain music-playing N64 ROM dumps, with all of the non-essential code dummied out, possibly with custom code compiled to handle setup and initialization, is a good way to go.
Streamed audio formats don't necessarily even need to be N64 related, and may be likely to end up in vgmstream instead.
Sequenced is hard enough to get right with the existing USF format. A more future proof USF format that doesn't rely on save states doesn't even need to require a new file format. The existing USF file format already supports both USFLIB and MINIUSF overlays containing either or both of ROM or save state. The save state is entirely optional in the file format, as is the ROM itself.
The problem is making the emulator library safely handle the case of no save state, and support cold booting a ROM image. I think that may be possible, it just needs a startup process that skips the state loading if there is no state to load.
Code coverage is also quite simple with the code I've assembled in the new lazyusf2 library. Unfortunately, it still requires the slower interpreter and full RSP interpreter modes, simply to make sure that every important piece of code or data is exposed and marked as used.
Also included in the binary array library that it uses to keep track of coverage, are simple functions for combining arrays, so that a caller that has already supplied a basic ROM image that already does full songs, and has a single uint32 somewhere that may be filled with an arbitrary song number, can run each known song for several loops, then combine all of the arrays together, zapping all of the unused bytes with zeroes.
I already wrote such a coverage tool for hand ripped 2sflib/mini2sf sets, since that Pokémon: Explorers Procyon Studios rip didn't work with the existing tools, so had to be trimmed through sheer brute force. It actually stops the coverage checking after so much memory coverage inactivity, or a period of play time, or so. I forget which.
This tool would need to be rewritten to trim usflib/miniusf sets, with the ripping tools already handling the part of writing a full set of proper song numbers. Or if you're feeling adventurous, write actual song data to the miniusf files, and modify the coverage and trimming process to separate coverage data into regions covered by the usflib and by the miniusf. Knock yourself out. :D
A nice thing about ROM-only is you should be able to track coverage with full dynarec, since you only need to look at explicit ROM access. Doesn't quite do the trick if you unpack stuff and load a RAM image, though.