7z archives optimization by Vitamin at 5:23 AM EST on November 21, 2018

As everbody knows, almost all the joshw collection is stored in solid 7z archives. Unfortunately, it's extremely heavy to perform file random access, especially for mobile devices.

I've made a tool to optimize collection of 7z files - GitHub

It processes all the input 7z files (or found 7z files in specified folders, processed recursively) in the next way:
1) if archive is solid and size of the biggest solid block with more than 1 file is more than --max-solid-block-size (3M by default) then archive is recompressed in the next way:
1a) if archive compression ratio is worse than --min-unpacked-ratio (0.9 by default) and compression delta is less than --max-nonpacked-space-loss (2M by default), then archive will be uncompressed
1b) else it's recompressed with --compression-level (7 by default), maximum filters and maximum solid block size equal to --max-solid-block-size
2) else file is untouched

This transformation dramatically improves archives' random access time without significant space loss. I've managed to process all the archives in my local copy (2sf dsf fmtowns gbs gsf hes kss ncd nsf pc psf smd spc ssf usf) and got about 3Gb used space decreasing, so it's worth doing:)

If you don't want to waste your CPU, you can download files from my copy (I can give a link if interested).

P.S. To clarify: I'm the author of ZXTune chiptunes player and joshw collections are the significant part of online catalogues for Android version. Sometimes users complaint about slow operations with 7z archives, so I decided to improve the situation:)
by Knurek at 12:55 PM EST on November 21, 2018
@datschge: Any way to have that run on the server? I don't fancy reuploading all the stuff *again*...
by hcs at 1:45 AM EST on November 22, 2018
If not on the joshw server itself, those with a login for hcs64 could rsync it down there and do the conversion probably pretty fast.
by Vitamin at 5:38 AM EST on November 22, 2018
A little tip to speedup conversion process:

$ mkdir temp
$ sudo mount -o rw,size=9G -t tmpfs none temp

Script uses ${CWD}/temp folder for extracted archives content, so better to put it on tmpfs for speedup and hdd/ssd saving:)

Some of the big files may not fit to specified space, so you should make second pass w/o tmpfs to retry all the failed conversions.
You need about 50G of free space at peak (22Gb for the biggest archive content and archive itself).

edited 5:39 AM EST November 22, 2018
by Squaresoft74 at 12:47 PM EST on November 22, 2018
Sorry for off topic, but since you're around i may ask if there is any reason for ZXTune not being available from Google Play Store for Nvidia Shield ATV users ?
by Vitamin at 2:38 PM EST on November 22, 2018
Because it's not adapted to work on Android TV devices. And I don't know will it someday or not.
by datschge at 10:09 AM EST on November 24, 2018
@Knurek: see mail.

@Vitamin: The script needs python 3. On python 2.7 (still default on Fedora/RedHat/CentOS) one needs to install and import from functools32 instead. Any idea how to cover both versions at the same time?
by Vitamin at 5:54 PM EST on November 24, 2018
@datschge: I'm not a real python developer, but I'll try.
According to distrowatch, latest stable fedora already contains python 3.7

Go to Page 0

Search this thread

Show all threads

Reply to this thread:

User Name Tags:

bold: [b]bold[/b]
italics: [i]italics[/i]
emphasis: [em]emphasis[/em]
underline: [u]underline[/u]
small: [small]small[/small]
Link: [url=http://www.google.com]Link[/url]


HCS Forum Index
Halley's Comet Software
forum source
Generated in 0.0058s;