Question about HTTP::Client

Help & Support
1

Hello, because actually now crystal is in 1.2.2 version, I have a question. In past, HTTP::Client didn’t have the fonctionnality to do automatically redirections. And actually ?

Because I would like to download a file with crystal and as well write a nice output when I’m downloading a file. For example showing percentage, amount of received data, progress bar …etc

2

I also don’t know if my information is up to date, but I’m pretty sure that HTTP::Client from standard library will not handle redirects automatically.

For a featureful HTTP client (that handles redirects) I recommend this one instead:

3

Sorry to come back again to my old post, but any news about the redirects solutions for crystal ? I saw HTTP:Client didn’t implement that actually. Is halite the only available library actually for that ?

If it’s not, just one question, how can I get the current network speed of download, amount of data transfered …etc ?

4

To handle redirects you can look at the response code and if it’s 301 or 302 then do another request to the location specified in the Location header.

For the download speed you could use Time.monotonic to calculate how much you download every second.

5

Hi ! Sorry I know I made this post long time ago.

But I’m coming back because I found an another shard to have an HTTP client with redirections supports, and more updated.

I’m using Crest: GitHub - mamantoha/crest: HTTP and REST client for Crystal

I did that to get the size of the downloaded data.

Is it correct in that way ? Because network isn’t my speciality (this code is just a test):

require "crest"

startingTime = Time.monotonic
currentTime = Time.monotonic

Crest.get("https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.13.12.tar.xz") do |resp|
  filename = resp.filename || "crystal.zip"

  while line = resp.body_io.gets
    if (currentTime - startingTime).seconds > 1
      text = "crystal.zip | Received data: #{line.bytesize} bytes/seconde"
      print text
      print "\r"
      startingTime = Time.monotonic
    end
    currentTime = Time.monotonic
  end

  File.open(filename, "w") do |file|
    IO.copy(resp.body_io, file)
  end
end

I have two questions as well. How can I get before my code download to get the downloaded filename and get the total size of the file ? For the filename, do I need to just extract it from the url ?
I would like as well to know how much data are already downloaded.

6

Hi @Fulgurance . Maintainer of crest here.

Regarding your question on how to retrieve the downloaded file’s filename and total size, you can do this by checking the response headers. Specifically, you can look for the “Content-Disposition” header to get the filename and the “Content-Length” header to get the total size.

If you also want to track the amount of data downloaded, you can use this code.

require "crest"

url = "https://github.com/crystal-lang/crystal/archive/1.8.1.zip"
buffer_size = 4096
downloaded_size = 0

Crest.get(url) do |response|
  output_file = response.filename || "crystal.zip"

  File.open(output_file, "w") do |file|
    buffer = Bytes.new(buffer_size)

    loop do
      bytes_read = response.body_io.read(buffer)

      break if bytes_read == 0

      file.write(buffer[0, bytes_read])

      downloaded_size += bytes_read

      print "Received data: #{downloaded_size.humanize_bytes}"
      print "\r"
    end
  end

  puts
  puts "Download complete! Saved as #{output_file}"
end
1 Like