Streaming Zip Files from S3

Help & Support
1

Hello all,

I’ve been interested in Crystal for a while, and am finally using it for a project at work! We have a virtual filesystem in the database but are storing the files in Wasabi (basically S3). The project is to create and stream a zip file on the fly.

I’ve looked at a LOT of zip packages that would allow us to add directories and Zip::Writer does that perfectly.

Anyways, we’ve got a working prototype that is mostly working. But when I try to zip a large number of files, sometimes the resulting zip is invalid (central directory is missing).

Given that I’m streaming this on-demand and simply storing the files (no compression), is there a way to check the integrity of the zip file? Or would that require creating the file locally on disk?

Also for reference, I’m starting the zip file like this:

 env.response.output << Compress::Zip::Writer.open(env.response.output) do |zip|

I’m adding the contents of the file like this:

 wasabi.get_object(bucket, wasabi_path) do |res|
   zip.add(file.as_h.dig("name").as_s, res.body_io)
 end

Lastly, is there anything about the way I’m adding contents of the zip file that could be a problem?

Thanks

2

I don’t suppose you’re able to create something that reproduces it outside of your code?

One thing that looks a bit strange is env.response.output << Compress::Zip::Writer.open, pretty sure this is writing nil to the response which probably isn’t what you want. No need to do that as the Zip would already be written to the IO you pass to .open. env.response is also already an IO, so I’d probably just write to that versus env.response.output.

3 Likes
3

If you create them locally do they sometimes lack an index? Maybe you can create and log an md5 to see if what you “think” is making it to wasabi isn’t…to me it smells like a lack of flushing somewhere…