-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Zip::Writer Arithmetic overflow when adding 1 million files #14534
Comments
Pretty sure this is a limit with ZIP archives themselves, in that they can only have 65,535 entries, which is the max value of a Ideally solution to this would be #7103. But we should at least throw a more informative error message I'd say. |
ok so if I understand correctly what you are saying is the ZIP writer within the Stdlib supports the traditional zip archive format which has a 65k limit on entries and so to go above this you need something like the 3rd party lib you mentioned? It does solve my immediately problem - so thank you for that. Interestingly, if I generate 1,000,000 4 byte files on disk and then run the zip command line tool over it, it will sit and think about it for a bit but it will compress the whole thing in the end. Am I confusing the functionality of CLI tool with what the underpinnings of the Zip format or zlib offer? In any case thank for the feedback. |
Yea the API docs for
So it's using the original non-ZIP64 version which has that limitation. There is #11396, but not sure of the status of that PR at the moment. |
Thanks for that. I tried out the WeTransfer/cr_zip_tricks and it did work on the isolated test project I made for exploring this issue, however it sadly gave inexplicable compile-time errors (some kind of inability to see the built-in Digest library) on the main project I wanted to use it with. Lacking the time to find out what the root cause was I found another library: https://shardbox.org/shards/zip64/releases/0.2.1/ |
Bug Report
While I was able to find existing issues for "Arithmetic overflow", I could not find any specific to the Zip Writer.
This problem was encountered while deploying a solution for compressing offsite backups for client data on a server.
Sample code to illustrate crash:
Result:
Environment:
Dev / Crystal installation:
Crystal 1.12.1 [4cea101] (2024-04-11)
LLVM: 15.0.7
Default target: aarch64-apple-macosx11.0
The text was updated successfully, but these errors were encountered: