Multithreaded extraction

[aknarts]

This is mostly just a question, I am working on extracting a 21.8G file which consist of 184k files.

I am running the base extraction in a thread pool but I am not able to do the extraction separately as the ZipFile is not Send. My hope with that was that the just one thread would be chewing through the one big file while others chew through the smaller files, but sadly this is not an option.

I also tried to open the archive in every thread but this proved to be a bad idea if not only for the slow down.

I am not sure what other options there are, I am pretty much using the code from examples for extraction. If anyone has some examples how to speed this up or how to best approach the problem I am all ears. Every single zip file I will be dealing with will be roughly the same specs so not concerned about some edge cases.

[NobodyXu]

I think you can do it like this:

• Open a ZipArchive
• Obtain number of files in the archive using ZipArchive::len
• Use rayon to distribute (0..len).into_par_iter().map(...) and also includes a clone of ZipArchive
• Use ZipArchive::by_index to obtain each file with zero parsing cost and extract them manually

Checkout ZipArchive::extract for how to extract each individual file.

[NobodyXu]

@aknarts And I strongly recommend you to use sth like mmarinus which loads the file using mmap.

This would avoid the read + write + seek syscall and the copying from kernel to userspace and also gives you userspace caching for free.

[adetaylor]

I've been working on this and it works pretty well. My organization has a somewhat long-winded open source release process which I'm going through, so in a few weeks I should have a published crate which does this.

[adetaylor]

Here it is! https://github.com/google/ripunzip

[Plecra]

Fantastic! I was looking for this one :)

[a1phyr]

For the record, I created sync_file, a crate that use OS-specific extensions to safely read a file concurrently from several threads without a mutex, which destroys parallelism (plus makes seeking essentially free).

If you are interested, have a look at https://docs.rs/sync_file/latest/sync_file/struct.SyncFile.html.

[Plecra]

Thanks for the feature request :) There's no question we should support this.

We can probably find a way to make ZipFile: Send but I'm not excited to try. I think this is another feature that's going to have to wait.