IO.copy buffer too small?

Crystal Contrib
21

@beta-ziliani Seems those are the ones I can find, which has relevancy and a hardcoded magic number buffer

./src/digest/digest.cr:222:    buffer = uninitialized UInt8[4096]
./src/http/web_socket.cr:14:    @buffer = Bytes.new(4096)
./src/http/client/response.cr:127:      line = io.gets(4096, chomp: true)
./src/io.cr:561:        buffer = uninitialized UInt8[4096]
./src/io.cr:851:    buffer = uninitialized UInt8[4096]
./src/io.cr:853:      read_count = read(buffer.to_slice[0, Math.min(bytes_count, 4096)])
./src/io.cr:863:    buffer = uninitialized UInt8[4096]
./src/io.cr:1155:    buffer = uninitialized UInt8[4096]
./src/io.cr:1179:    buffer = uninitialized UInt8[4096]
./src/compiler/crystal/tools/playground/server.cr:194:            output = String.new(4096) do |buffer|
./src/compiler/crystal/tools/playground/server.cr:195:              length = io.read_utf8(Slice.new(buffer, 4096))
./src/crystal/system/wasi/dir.cr:6:    property buf = Bytes.new(4096)
./src/io/buffered.cr:14:  @buffer_size = 8192
./src/io/encoding.cr:62:    BUFFER_SIZE     = 4 * 1024
./src/io/encoding.cr:63:    OUT_BUFFER_SIZE = 4 * 1024
22

Even when monkey-patching all of those things, I still see reading of only 4096 :crazy_face:

What am I missing :sob:

1 Like
23

Could you provide a working code?

24

Not all of those are generic buffers. The length limit on parsing the HTTP status line for example is explicitly chosen as 4K to have somewhat reasonable limit. No genuine status line is going to exceed that.

The most relevant ones are certainly IO.copy and other IO method, as well as IO::Buffered. Maybe websocket and digest as well. But the rest is probably no concern.

25

I have created a PR to introduce a constant for the default size and increase it to 32K: Introduce `IO::DEFAULT_BUFFER_SIZE` by straight-shoota · Pull Request #12507 · crystal-lang/crystal · GitHub

4 Likes