Working with large bin files, proper way to do it?

Help & Support
1

I’m working with some large .bin files, and I was wondering if anyone had any tips for dealing with them in Crystal. I’ve been having trouble getting them to read right, and I need to replace a section of it’s bytes with another section. So far even copying the file byte by byte isn’t working correctly.

IPCAM_SH_START = 0xFD444
IPCAM_SH_END = 0xFD580
FWVERSION_BIN_START = 0x13914
FWVERSION_BIN_END = 0x139DA
UPDATE_ORIGINAL_FILE = "client/rsrc/update_original"

update = File.open(UPDATE_ORIGINAL_FILE)
io = IO::Memory.new
new_file = File.open("client/rsrc/new_update", "w+")
byte_counter = 0
loop do
  bytes_to_read = update.read_byte
  break if bytes_to_read.nil?
  byte_counter += bytes_to_read
  IO.copy(update, new_file, bytes_to_read)
end

puts "Total bytes #{byte_counter}"

puts "New File bytes #{File.size("client/rsrc/new_update")}"

puts "Old File bytes #{File.size(UPDATE_ORIGINAL_FILE)}"

Which all give me wildly different results:

Total bytes 1029743

New File bytes 1024199

Old File bytes 1037750

[Done] exited with code=0 in 0.868 seconds

Anyone know what I’m doing wrong.

2

It’s IO.copy(source, destination, limit : Int). So you are passing IO.copy(update, new_file, read_byte). Note that update.read_byte returns a single byte, and you are passing that as limit, which doesn’t do what you need.

You can probably do:

byte = update.read_byte
break unless byte

byte_counter += 1
new_file.write_byte(byte)

But you probably want to copy the entire file. For that you are replicating what IO.copy does, though IO.copy will probably overflow for large files. Just take a look at what IO.copy does but start with count = 0_u64 or similar.

(we should probably update that method to return a UInt64, it probably makes more sense)

1 Like