Kemal upload slow

Help & Support
1

Hi, I made a simple endpoint to upload a file

post "/upload" do |env|
  parse_multipart(env) do |f|
    file_path = File.join Dir.current, "filesaved"
    File.open(file_path, "w") do |file|
      IO.copy(f.data, file)
    end
    "Upload complete"
  end
end

But when I upload something using curl, from and to my local machine the upload speed is not higher than 10 Mb/s. Is it related to kemal or am I missing something ?

2 
curl -X POST  -F "image1=@/tmp/file4" http://localhost:3000/upload -o test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4096M    0     0  100 4096M      0  8723k  0:08:00  0:08:00 --:--:-- 8386k
1 Like
3

Could you please give a complete working code example?

4 
require "kemal"
require "json"
Kemal.config.public_folder =  "/home/x"

post "/upload" do |env|
  parse_multipart(env) do |f|
    file_path = File.join Dir.current, "filesaved"
    File.open(file_path, "w") do |file|
      IO.copy(f.data, file)
    end
    "Upload complete"
  end
end

Kemal.run
5

This might be https://github.com/crystal-lang/crystal/pull/11242

Maybe some day someone will pick it up…

1 Like
6

It is precisely this, I just checked. With this code:

require "http"

http = HTTP::Server.new do |context|
  MIME::Multipart.parse context.request do |headers, io|
    file_path = "public"
    File.open(file_path, "w") do |f|
      IO.copy(io, f)
    end
  end
end

http.listen 3000

Using the latest release of Crystal, I uploaded a ~2GB file to it, and it transfers at 162MB/sec, taking about 11 seconds:

➜  upload_demo git:(main) curl -X POST  -F "image1=@file" http://localhost:3000/upload -o test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 56 1839M    0     0   56 1034M      0   162M  0:00:11  0:00:06  0:00:05  162M

With your patch, it transfers at 1.7GB/sec in barely over a second, over 10x as fast:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1839M    0     0  100 1839M      0  1678M  0:00:01  0:00:01 --:--:-- 1685M

EDIT: I’d forgotten to compile the app in release mode.

7 Likes
7

sorry, noob question but how do I apply the patch ?

8

I made a gist with (I believe) all the updated methods from the PR. You can load it into your app by adding this to your shard.yml:

dependencies:
  io-delimited-optimized:
    git: https://gist.github.com/jgaskins/91186b3451988f714b3bebe45b63e29d

And then in your code, just add require "io-delimited-optimized". Then enjoy faster upload processing in your app.

4 Likes
9

thank you very much!

10

Wired, why no one mention, those code never work for latest version kemal + crystal? no parse_multipart method exists at all.

Following code come from kemal guide works, and speed is not slow than the questioner.

require "kemal"

post "/upload" do |env|
  HTTP::FormData.parse(env.request) do |upload|
    filename = upload.filename
    # Be sure to check if file.filename is not empty otherwise it'll raise a compile time error
    if !filename.is_a?(String)
      puts "No filename included in upload"
    else
      file_path = ::File.join [Kemal.config.public_folder, "uploads/", filename]
      File.open(file_path, "w") do |f|
        IO.copy(upload.body, f)
      end
      puts "Upload ok"
    end
  end

  env.redirect "/index.html"
end

Kemal.run
11

It’s easy to overlook small details like that when trying to reduce your reproduction of a problem. And when I looked at the Kemal docs for uploading files and it looked nearly identical, I figured that’s what Paulo was getting at and used that code in my original benchmark before peeling back layers down to MIME::Multipart.

It only added about 30 seconds to my investigation. It would’ve taken more time to come to the forum to ask for clarification.

12

Hi, the link you reference is not work for me, because env.params.files always empty.

i guess those code was abandoned, the working code is here

13

I have no idea whether the request completes successfully with that code (I didn’t wait for it to finish), but we don’t need a successful request to take note of the upload speed. It can be observed before env.params.files completes.

14

Yes, you are right, the patch is really impressive for the upload speed improve.

but, for the original question, d is below 10 Mb/s on local machine i thought the issue is come from different things, e.g. use old version kemal/crystal ?