The Crystal Programming Language Forum

Working with large bin files, proper way to do it?

I’m working with some large .bin files, and I was wondering if anyone had any tips for dealing with them in Crystal. I’ve been having trouble getting them to read right, and I need to replace a section of it’s bytes with another section. So far even copying the file byte by byte isn’t working correctly.

UPDATE_ORIGINAL_FILE = "client/rsrc/update_original"

update =
io =
new_file ="client/rsrc/new_update", "w+")
byte_counter = 0
loop do
  bytes_to_read = update.read_byte
  break if bytes_to_read.nil?
  byte_counter += bytes_to_read
  IO.copy(update, new_file, bytes_to_read)

puts "Total bytes #{byte_counter}"

puts "New File bytes #{File.size("client/rsrc/new_update")}"

puts "Old File bytes #{File.size(UPDATE_ORIGINAL_FILE)}"

Which all give me wildly different results:

Total bytes 1029743

New File bytes 1024199

Old File bytes 1037750

[Done] exited with code=0 in 0.868 seconds

Anyone know what I’m doing wrong.

It’s IO.copy(source, destination, limit : Int). So you are passing IO.copy(update, new_file, read_byte). Note that update.read_byte returns a single byte, and you are passing that as limit, which doesn’t do what you need.

You can probably do:

byte = update.read_byte
break unless byte

byte_counter += 1

But you probably want to copy the entire file. For that you are replicating what IO.copy does, though IO.copy will probably overflow for large files. Just take a look at what IO.copy does but start with count = 0_u64 or similar.

(we should probably update that method to return a UInt64, it probably makes more sense)

1 Like