The issue seems to be that IO::Memory stores its capacity as an Int32, so once youâve read 2GB into the buffer, its size overflows to become a negative number. I donât know if this is the exact problem youâre experiencing, but it seems pretty likely.
With very few exceptions, I doubt anyone actually needs to hold that much data in a single buffer, and it looks like you can avoid holding the entire output in memory by using IO.pipe instead of an IO::Memory and passing a block to Process.run:
reader, writer = IO.pipe
Process.run "cat xx.txt|cut -f 1-6|sort -k 1,1", shell: true, output: writer do |process|
until process.terminated?
line = reader.gets
puts line
end
end
Notice this code creates two IOs with IO.pipe â we tell the child process to write to one end and our code reads from the other end.
IO::Memory isnât a good fit for streaming both input and output at the same time, especially across processes since it only has a single position marker for both reading and writing â that is, if you call io.puts "foo", calling io.gets wonât read "foo" since your position in the buffer is at the end of the buffer you just wrote. It also doesnât release memory after itâs been read. Thatâs a feature (itâs what allows io.rewind), it just doesnât scale well for huge inputs. :-D
Really thanks a lot~ output.gets do works well! My code now is like this:
ifile = ARGV[0]
Process.run("cat #{ifile}|cut -f 1-6|sort -k 1,1", shell: true) do |proc|
while line = proc.output.gets
puts line
end
puts "here you get end the file"
end
proc.output.gets will get the real terminal of file at last.
Still, any reason not to convert this little command to pure Crystal? If the actual code is really like your example, it will be trivial to do - and much more efficient (and robust).
Depends on whether thatâs needed. If this is invoked thousands of times per second by an automated process or by a network request, that would be a great reason to pull that functionality into Crystal. If itâs only ever run by hand, the difference in efficiency will never be noticed on anything resembling modern hardware.
yes, my actual code is like that. If the perfromance matter, I will try to write pure Crystal code.
At least I learned the way about read the ouput line by line from system command call.