Empty content when reading 'big' packets on TCP/IP

Help & Support
1

Every time I try to read a relative big packet bigger than actual window size
I got empty data after a TCP/IP is updating it’s window size.

The content after sync is zero filled but the size is what expected
but the content before is ok and error indicating from OS.

I use a WINSHARK on the receiving side so I can see data is properly delivered
to the consumer machine.

Background. The data is a resultset from a MySql-server collected into big packets
by TLS. It seems like TSL will packet things into max 2**14 (16,384 bytes)

If I work around 5kb (by elaborate with SQL LIMIT) which often is the startup window-size
in TCP/IP I can see that I all data from first window is correct but the second window is delivered but zero but.

In any case one must be able to read a TCPIPsocket with bigger size than a window-size, and a window size could size of one TCP/IP packet!

Or am I stupid? Someone else seen behavior and solved it.
Or a bug somewhere?
Or some settings for TCPSocket?

      @socket = TCPSocket.new(@mysql_options.host, @mysql_options.port)
      @socket.recv_buffer_size=17000

Here is a pocket of 16kb

BRIDGETls::read_packet_result_set. Try local_payload_size=16401

which fails with zero content data after first tcp window

even with quit small packet size it will fail 50% of run cases

BRIDGETls::read_packet_result_set. Try local_payload_size=6207
2

Embarrassing!!
Should have know better. At my age

Old code was

local_payload_size = given_by_tls_header
local_payload = Bytes.new(local_payload_size)

@socket.read(local_payload)
...

Did not care, did not know about @socket.read returned size

New working code

# same intro
local_payload_size = given_by_tls_header
local_payload = Bytes.new(local_payload_size)

got_size = @socket.read(local_payload)
while got_size < local_payload_size
  size_this_read = @socket.read(local_payload[got_size..-1])
  got_size += size_this_read
end
...
1 Like
3

Oof, I’ve run into this a few times, too. You can also use IO#read_fully, which blocks until the full slice is read. It’s pretty much exactly what you’ve got in your while loop, so your diff would look like this:

-@socket.read(local_payload)
+@socket.read_fully(local_payload)
3 Likes
4

There is also an active PR for adding IO#read_greedy which is more convenient than #read_fully if you don’t expect a fixed size: https://github.com/crystal-lang/crystal/pull/16535
Unfortunately, it’ll only be available in Crystal 1.20.
You could monkey-patch it into your code base, though.

And I’ve been wondering whether we could reduce surprises like this by changing the behaviour of the default #read method: https://github.com/crystal-lang/crystal/issues/14605#issuecomment-3695047933
So this is a good example where that would’ve been helpful.

5

Thanks for your generous engagements!

All works fine and nicer now when I use read_fully instead of looping around read.

Thanks again

1 Like