IO.copy(src, dst) handles the back pressure?

Hello,

I’m doing a small proxy, I don’t know if I risk overflowing RAM when downloading large files because the browser is slower to download the file via Internet than the server via internal network (because the bandwidth).

Does IO.copy handle back pressure?
I have the impression that yes because I made some tests of simultaneous download, the memory seems ok. Also the data goes through IO::Memory and a small buffer (4kb) but I may be wrong?

It’s just for one endpoint, if IO.copy handles it well it saves me from putting a Nginx proxy in front of it just for this case.

HTTP::Client.get(url) do |res|
        if res.status_code != 200
          Log.debug { "#{res.status_message}: #{url}" }
          return
        end

        ctx.response.headers.merge!(res.headers)
        IO.copy(res.body_io, ctx.response.output) if res.body_io?
      end

In this case IO.copy handles the back pressure?

Hi!

I don’t know what “back pressure” means in this case, but here’s IO.copy source code:

It allocates a small buffer on the stack, reads from the source IO to that buffer, and then writes from that buffer to the target IO. So this doesn’t involve heap memory allocations at all.

Thanks.
The source is res.body_io, if it is read (copied to response.output) too slowly, do you think res.body_io will grow (in RAM) or is it limited via a buffer too?

1 Like

Yes, body_io is limited by a buffer. In general when using IO in Crystal there are no memory allocations involved.

2 Likes

This is great, thank you :+1:

For info, I’ve done a lot of tests with 250MB files in parallel, it seems very stable, the memory grows and shrinks during uploads. It does the job. In case I’m not looking well or some other contraindication, I prefer to ask :slight_smile:

2 Likes