Upload image failed use HTTP::Client but test with postman work

zw963 · June 19, 2025, 4:54am

Following is my code (I need form_data to upload a file, but for now, test with a source: "https://image_url" is enough.

post "/api/upload" do
    # file_path = params.from_multipart.last["source"].path

    io = IO::Memory.new

    form_data = HTTP::FormData::Builder.new(io)
    form_data.field("key", "my_key")
    form_data.field("source", "https://avatars.githubusercontent.com/u/549126?s=96&v=4")

    response = HTTP::Client.post(
      url: "https://freeimage.host/api/1/upload",
      headers: HTTP::Headers{"Content-Type" => "multipart/form-data"},
      form: io
    )

    pp! response
    if response.success?
      image_url = JSON.parse(response.body).dig("image", "display_url")
      json({image_url: image_url}, HTTP::Status::OK)
    else
      json(HTTP::Status::BAD_REQUEST)
    end
  end

When I upload, I always get error like this:

{\"status_code\":400,\"error\":{\"message\":\"Invalid API v1 key.\",\"code\":100},\"status_txt\":\"Bad Request\"}"

But if i test use a tool like postman, it works (check following screenshot)

Following is the screenshot of API page

I have another question.

Either a image URL or a base64 encoded image string. You can also use FILES[“source”] in your request.

What is the means of FILES["source"] in your request, how to do that use HTTP::FormData::Builder?

thanks

lasso · June 19, 2025, 1:44pm

Docs (HTTP::FormData::Builder - Crystal 1.16.3) says that the finish method must be called to finalize the form data.

Using HTTP::FormData.build() with a block instead of using HTTP::FormData::Builder.new() seems to call finish automatically at the end of the block, so maybe that is a better way.

Blacksmoke16 · June 19, 2025, 2:30pm

Also may need to do like form: io.rewind to ensure it starts reading from the beginning of the IO.

zw963 · June 19, 2025, 5:10pm

Wired, after apply both changes, still not work.

post "/api/upload" do
    # file_path = params.from_multipart.last["source"].path

    io = IO::Memory.new

    HTTP::FormData.build(io) do |form_data|
      form_data.field("key", "my_key")
      form_data.field("source", "https://avatars.githubusercontent.com/u/549126?s=96&v=4")
    end

    response = HTTP::Client.post(
      url: "https://freeimage.host/api/1/upload",
      headers: HTTP::Headers{"Content-Type" => "multipart/form-data"},
      form: io.rewind
    )

    pp! response
    if response.success?
      image_url = JSON.parse(response.body).dig("image", "display_url")
      json({image_url: image_url}, HTTP::Status::OK)
    else
      json(HTTP::Status::BAD_REQUEST)
    end
  end

Following is the response output:

web          | response # => #<HTTP::Client::Response:0x7f56eab4fc30
web          |  @body=
web          |   "{\"status_code\":400,\"error\":{\"message\":\"Invalid API v1 key.\",\"code\":100},\"status_txt\":\"Bad Request\"}",
web          |  @body_io=nil,
web          |
web          |  @cookies=nil,
web          |  @headers=
web          |   HTTP::Headers{"CF-RAY" => "95249ca8a8a70399-HKG",
web          |    "Cache-Control" => "no-cache, must-revalidate",
web          |    "Cf-Cache-Status" => "DYNAMIC",
web          |    "Connection" => "keep-alive",
web          |    "Content-Type" => "application/json; charset=UTF-8",
web          |    "Date" => "Thu, 19 Jun 2025 17:05:34 GMT",
web          |    "Expires" => "Thu, 19 Nov 1981 08:52:00 GMT",
web          |    "Last-Modified" => "Thu, 19 Jun 2025 17:05:34GMT",
web          |    "Nel" =>
web          |     "{\"report_to\":\"cf-nel\",\"success_fraction\":0.0,\"max_age\":604800}",
web          |
web          |    "Pragma" => "no-cache",
web          |    "Report-To" =>
web          |     "{\"group\":\"cf-nel\",\"max_age\":604800,\"endpoints\":[{\"url\":\"https://a.nel.cloudflare.com/report/v4?s=gOaUwEYuOLapkef4w4cfodQzHx9nu%2BDxzWvpV5C1BQhcvyn4WiQhxvzIYjbWvq8a8uVxcU%2BOfYXlmAMH5V4Zc2eRlHs%2BpZ%2FGc82fU0v1Wn%2F189T5am5Fr5iw\"}]}",
web          |
web          |    "Server" => "cloudflare",
web          |    "Set-Cookie" => "PHPSESSID=flotqnvatdlnj5gka8hnp03q7f; Path=/",
web          |    "Transfer-Encoding" => "chunked",
web          |    "X-Frame-Options" => "DENY",
web          |    "alt-svc" => "h3=\":443\"; ma=86400"},
web          |
web          |  @status=HTTP::Status::BAD_REQUEST,
web          |  @status_message="Bad Request",
web          |
web          |  @version="HTTP/1.1">

I try change form: io.rewind to body: io.rewind, no luck.

Any idea? thanks

zw963 · June 19, 2025, 5:20pm

I write a reproducable example.

require "http"
require "json"

io = IO::Memory.new

HTTP::FormData.build(io) do |form_data|
  form_data.field("key", "my_key")
  form_data.field("source", "https://avatars.githubusercontent.com/u/549126?s=96&v=4")
end

response = HTTP::Client.post(
  url: "https://freeimage.host/api/1/upload",
  headers: HTTP::Headers{"Content-Type" => "multipart/form-data"},
  form: io.rewind
)

body = JSON.parse(response.body)

pp! body

{"status_code" => 400,
   "error" => {"message" => "Invalid API v1 key.", "code" => 100},
   "status_txt" => "Bad Request"}

But, if test with postman, it work.

Blacksmoke16 · June 19, 2025, 5:39pm

Okay, I spent some time looking into this and there are a few things going on:

Are mixing application/x-www-form-urlencoded and multipart/form-data content types
In the latter case, not including the boundary in the header

When you pass something to the form property, it internally overrides your Content-Type header to application/x-www-form-urlencoded, yet the body of the request is multipart/form-data. You can reproduce this in Postman too by unchecking the default Content-Type header and adding a custom one with application/x-www-form-urlencoded.

Ultimately the fix is to settle on one or the other:

# application/x-www-form-urlencoded
response = HTTP::Client.post(
  url: "https://freeimage.host/api/1/upload",
  form: {
    "key"    => "my_key",
    "source" => "https://avatars.githubusercontent.com/u/549126?s=96&v=4",
  }
)

However, because you said you’re going to be uploading files, will likely want to use multipart/form-data. Which you’d want to handle by just passing it as the body. E.g. body: io.rewind. But there is still one thing missing. The content-type headers needs the boundary for multipart requests. Which you can get from the HTTP::FormData::Builder instance:

io = IO::Memory.new

form_data = HTTP::FormData::Builder.new(io)
form_data.field("key", "my_key")
form_data.field("source", "https://avatars.githubusercontent.com/u/549126?s=96&v=4")
form_data.finish

response = HTTP::Client.post(
  url: "https://freeimage.host/api/1/upload",
  headers: HTTP::Headers{"content-type" => form_data.content_type},
  body: io.rewind
)

zw963 · June 19, 2025, 6:07pm

Thanks very much for awesome explain.

Following is my final uploading image file directly version.

require "http"
require "json"

io = IO::Memory.new

file = File.open("rainbow.png")
form_data = HTTP::FormData::Builder.new(io)
form_data.field("key", "my_key")
form_data.file("source", file, HTTP::FormData::FileMetadata.new(filename: "rainbow.png"))
form_data.finish
file.close

response = HTTP::Client.post(
  url: "https://freeimage.host/api/1/upload",
  headers: HTTP::Headers{"content-type" => form_data.content_type},
  body: io.rewind
)

body = JSON.parse(response.body)

pp! body

It works! https://iili.io/FoQ92Xs.png

aiac · June 19, 2025, 6:47pm

Is there any way to avoid coping file data into memory?

zw963 · June 19, 2025, 6:52pm

My understanding , it’s just copying the IO object from file to form_data?

correct me if I am wrong.

aiac · June 19, 2025, 6:56pm

It tell me it would copy …

github.com/crystal-lang/crystal

src/http/formdata/builder.cr

63650d76a


      
          
            # We don't add a crlf before the first boundary if this is the first body part.
            @io << "\r\n" unless @state.start?
            @io << "--" << @boundary
            headers.each do |name, values|
              values.each do |value|
                @io << "\r\n" << name << ": " << value
              end
            end
            @io << "\r\n\r\n"
            IO.copy(io, @io)
          
            @state = State::FIELD
          end
          
          # Finalizes the multipart message, this method must be called before the
          # generated multipart message written to the IO is considered valid.
          def finish : Nil
            case @state
            in .start?
              fail "Cannot finish form-data: no body parts"

zw963 · June 19, 2025, 7:10pm

I guess I misunderstood something. I thought IO.copy not take too much memory, it just copies from one IO to another IO.

aiac · June 19, 2025, 7:17pm

Yeah, but the copy target is IO::Memory.

Maybe BodyType of post method should add a type BodyIOConsumer. then we could obtain client body socket io as copy target rather than IO::Memory.

Eidt

However, this may require the caller to provide an accurate content-length header (or chunked transfer). Perhaps this is why the standard library does not implement this feature.

jgaskins · June 21, 2025, 9:45pm

You can use IO.pipe to avoid loading the full file contents in memory. In a separate fiber (this is important, see below), serialize the FormData to the writer, then pass the reader to HTTP::Client#post:

require "http/client"
require "http/formdata"

reader, writer = IO.pipe
filename = "/path/to/your/file"
url = "http://your.url.here"
headers = HTTP::Headers.new

spawn do
  File.open filename do |file|
    HTTP::FormData.build writer do |form|
      headers["Content-Type"] = form.content_type
      form.file "source", file, HTTP::FormData::FileMetadata.new(filename: filename)
    end
  end
ensure
  writer.close
end

response = HTTP::Client.post url,
  headers: headers,
  body: reader

puts response.body

gets

On my machine, I’m uploading a 2.5GB file to a locally hosted server and only using 3.1MB to do it.

To validate that the entire file is being uploaded, I had the server return the file size in the body:

Total bytes uploaded: 2.49GiB

Important notes:

You must perform the write in another fiber. If you don’t, the pipe will block forever any time the serialized output (including HTTP and FormData overhead) exceeds your operating system’s pipe buffer. Performing the write in a separate fiber will allow the reads (by HTTP::Client) and writes (by HTTP::FormData) to be performed concurrently.
You must close the writer, so wrap that in an ensure block. If the writer is not closed, the reader will block forever waiting for the end of the stream.
Both of those blocking-forever situations may leak memory, depending on what your application does, and will definitely create a bad user experience.

aiac · June 22, 2025, 10:01am

This approach is also used in go lang, but I personally don’t like it very much.

	pr, pw := io.Pipe()
	defer pr.Close()

	mw := multipart.NewWriter(pw)
	content_type := mw.FormDataContentType()

	go func() {
		defer pw.Close()
		defer mw.Close()
		const bufferSize = 1024 * 8
		err := writeFilePart(mw, "image", file_path, bufferSize)
		if err != nil {
			slog.Error("Error on writing file part", "err", err)
		}
	}()

jgaskins · June 22, 2025, 1:01pm

Agreed, I’d love an easier way to do that because it’s too easy to get this wrong. I just don’t know what it looks like. I feel like it’d have to be less imperative. I’ll start a GitHub issue for it.

zw963 · June 22, 2025, 5:09pm

Thanks for you awesome answer.

I have adopted your suggestion, but made some minor changes, because original code assign the headers["Content-Type"] in the spawn, this assign value not work on my local, make the headers is empty, so, I use HTTP::FormData.new(writer) instead.

github.com/crystal-china/website

src/actions/api/upload.cr

1cc2b99cc

class Api::Upload < ApiAction
  post "/api/upload" do
    source = params.from_multipart.last["source"]

    # use IO.pipe instead of IO::Memory to reduce memory usage.
    # https://forum.crystal-lang.org/t/upload-image-failed-use-http-client-but-test-with-postman-work/8171/13

    reader, writer = IO.pipe
    headers = HTTP::Headers.new
    form_data = HTTP::FormData::Builder.new(writer)
    headers["Content-Type"] = form_data.content_type

    spawn do
      file = File.open(source.path) do |file|
        form_data.field("key", FREEIMAGE_HOST_API_KEY)
        form_data.file("source", file, HTTP::FormData::FileMetadata.new(filename: source.filename))
        form_data.finish
      end
    ensure
      writer.close

This file has been truncated. show original

Please point it out if there is a better solution.

Topic		Replies	Views
HTTP post a file Help & Support	2	362	February 1, 2021
Adding helpers to HTTP::Context to handle uploads and params Help & Support	4	419	November 12, 2020
POST request parsing error: could not find boundary in Content-Type Help & Support	5	312	March 30, 2023
IO+ vs IO from a request.body Help & Support	2	438	January 5, 2019
[solved] Cannot handle exception MIME::Multipart::Error Help & Support	7	311	July 27, 2023

Upload image failed use HTTP::Client but test with postman work

Eidt

Related topics