7z archives optimization by Vitamin at 5:23 AM EST on November 21, 2018
Hello!
As everbody knows, almost all the joshw collection is stored in solid 7z archives. Unfortunately, it's extremely heavy to perform file random access, especially for mobile devices.
I've made a tool to optimize collection of 7z files - GitHub
It processes all the input 7z files (or found 7z files in specified folders, processed recursively) in the next way: 1) if archive is solid and size of the biggest solid block with more than 1 file is more than --max-solid-block-size (3M by default) then archive is recompressed in the next way: 1a) if archive compression ratio is worse than --min-unpacked-ratio (0.9 by default) and compression delta is less than --max-nonpacked-space-loss (2M by default), then archive will be uncompressed 1b) else it's recompressed with --compression-level (7 by default), maximum filters and maximum solid block size equal to --max-solid-block-size 2) else file is untouched
This transformation dramatically improves archives' random access time without significant space loss. I've managed to process all the archives in my local copy (2sf dsf fmtowns gbs gsf hes kss ncd nsf pc psf smd spc ssf usf) and got about 3Gb used space decreasing, so it's worth doing:)
If you don't want to waste your CPU, you can download files from my copy (I can give a link if interested).
P.S. To clarify: I'm the author of ZXTune chiptunes player and joshw collections are the significant part of online catalogues for Android version. Sometimes users complaint about slow operations with 7z archives, so I decided to improve the situation:)
Script uses ${CWD}/temp folder for extracted archives content, so better to put it on tmpfs for speedup and hdd/ssd saving:)
Some of the big files may not fit to specified space, so you should make second pass w/o tmpfs to retry all the failed conversions. You need about 50G of free space at peak (22Gb for the biggest archive content and archive itself).
edited 5:39 AM EST November 22, 2018
by Squaresoft74 at 12:47 PM EST on November 22, 2018
Sorry for off topic, but since you're around i may ask if there is any reason for ZXTune not being available from Google Play Store for Nvidia Shield ATV users ?
@Vitamin: The script needs python 3. On python 2.7 (still default on Fedora/RedHat/CentOS) one needs to install and import from functools32 instead. Any idea how to cover both versions at the same time?