Download a file

Hello, actually I’m writing a package manager for Linux with Crystal.
I have just one problem, I don’t understand how to download a file with crystal.

For example if I want to download this archive:https://download.savannah.gnu.org/releases/acl/acl-2.3.1.tar.xz

How I can do ?

I’m assuming you just want to save the file to the file system? If so it would be something like:

require "http"

HTTP::Client.get "https://download.savannah.gnu.org/releases/acl/acl-2.3.1.tar.xz" do |response|
  File.open("out.tar.xz", "w") do |file|
    IO.copy response.body_io, file
  end
end

However, HTTP::Client doesn’t support redirects at the moment, see HTTP::Client: follow redirects · Issue #2721 · crystal-lang/crystal · GitHub, so you will probably have to add in some extra logic to handle that case. I.e. check for response.status.redirection? and if true, look for the location header and make another request before copying the contents to the file.

Hi again, sorry I know it’s a very old post.

So I followed your example, and I tried with your explanation to have the expected result.

require "http"

HTTP::Client.get("https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.13.12.tar.xz") do |response|

    if response.status.redirection?
        HTTP::Client.get(response.headers["location"]) do |response|

            File.open("linux-5.13.12.tar.xz", "w") do |file|
                IO.copy response.body_io, file
            end

        end
    else
        File.open("linux-5.13.12.tar.xz", "w") do |file|
            IO.copy response.body_io, file
        end
    end

end

I am not very good actually with networking :zipper_mouth_face:

I am definitely sure this can be improve a lot, but I think I didn’t understand everything.

If I use this code, is it okay or it’s a bit slow ?
Like that can I handle multiple redirections ?

How can I mesure received and sended data amount ? Because I would like to provide a nice cli output

I haven’t tried this shard but it looks like it can handle redirects and there’s an example to download a file:

rewrite your code to

require "http/client"

def download_with_redirect(download_link : String, save_path : String = File.basename(download_link), redirect_countdown : Int32 = 1)
  HTTP::Client.get(download_link) do |response|
    if response.status.success?
      File.open(save_path, "w") { |file_io| IO.copy response.body_io, file_io }
    elsif !response.status.redirection?
      raise "error response"      # define your error handling here
    elsif redirect_countdown > 0 # follow redirect
      new_link = response.headers["localtion"]
      download_with_redirect(new_link, save_path, redirect_countdown - 1)
    else
      raise "too many redirection"
    end
  end
end

download_with_redirect("https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.13.12.tar.xz", "linux-5.13.12.tar.xz", 5) # 5 is number of redirection attempts

and it should handle redirects correctly.

1 Like

Oh thanks a lot. How can I calculate the received amount data ? How I now how much data I received ?

A simple way is to just use an intermediate buffer, where you read X bytes from the server, then write X bytes to the destination.

Here’s a whole example client that lets you download something over HTTP(S) to a file and handles redirects.

require "http/client"

abort "No URL" unless ARGV.size > 0
abort "No output file" unless ARGV.size == 2

# Don't clobber files.
abort "File exists" if File.exists?(ARGV[1])

url = ARGV[0]
downloaded = false

# Loop until we've downloaded something (or raise an exception and die spectacularly).
until downloaded
  # Do a GET to the url
  HTTP::Client.get(url) do |resp|
    # Check for a redirect
    if resp.status.redirection?
      # Get a new URL
      if location = resp.headers["location"]
        url = location
        puts "Redirected to #{url}"
      else
        abort "Got status #{resp.status_code}, but no location was sent"
      end

      break # Get out of the .get
    end

    # Check for status 200
    if resp.status_code == 200
      # We'll use a buffer so that we can print a percentage.
      buf = Bytes.new(65536)
      totalRead = 0
      len = 0

      # If the headers have a Content-Length, then we can provide a
      # percentage.
      if resp.headers["Content-Length"]?
        len = resp.headers["Content-Length"].to_i32
      end

      # Open the output file
      File.open(ARGV[1], "wb") do |file|
        # Start reading the data from the HTTP client response.
        while (pos = resp.body_io.read(buf)) > 0
          file.write(buf[0...pos])

          # Print the progress
          if len > 0
            totalRead += pos
            puts "Downloaded: #{totalRead} / #{len}"
          else
            totalRead += pos
            puts "Downloaded: #{totalRead} / ???"
          end
        end
      end

      downloaded = true
    else
      # Don't handle any others.
      abort "Received HTTP status code #{resp.status_code}"
    end
  end
end
3 Likes

Wow nice, thanks a lot.

1 Like

So I guess in this code the pos variable is the amount of downloaded data ?

I would like to print the download data per second

you can use Time.monotonic to measure time duration between each write loop
the rest is just math.

it will be more accurate if you use channel to print the status every 1 seconds, but the code will be more complex.

Thanks but to be honest I prefer to use the standard API

I added this to one of my programs just last night. Taking the code from before…

require "http/client"

abort "No URL" unless ARGV.size > 0
abort "No output file" unless ARGV.size == 2
abort "File exists" if File.exists?(ARGV[1]) # Don't clobber files.

url = ARGV[0]
downloaded = false

# Loop until we've downloaded something (or raise an exception and die spectacularly).
until downloaded
  # Do a GET to the url
  HTTP::Client.get(url) do |resp|
    # Check for a redirect
    if resp.status.redirection?
      # Get a new URL
      url = resp.headers["location"]? || abort "Got status #{resp.status_code}, but no location was sent"
      puts "Redirected to #{url}"
      break # Get out of the .get
    end

    lastSpeedUpdate = Time.monotonic

    # Used to hold the average speed
    avg = 0

    # The number of bytes we've read between updates.
    bytesLastPeriod = 0

    # Check for status 200
    if resp.status_code == 200
      # We'll use a buffer so that we can print some progress.
      buf = Bytes.new(65536)
      totalRead = 0
      len = 0

      # If the headers have a Content-Length, then we can provide a
      # percentage.
      len = resp.headers["Content-Length"].to_i32 if resp.headers["Content-Length"]?

      # Open the output file
      File.open(ARGV[1], "wb") do |file|
        # Start reading the data from the HTTP client response.
        while (pos = resp.body_io.read(buf)) > 0
          # Calculate average speed, one per second.
          lapsed = Time.monotonic - lastSpeedUpdate
          if lapsed.total_seconds >= 1
            div = lapsed.total_nanoseconds / 1_000_000_000
            avg = (bytesLastPeriod / div).to_i32!
            bytesLastPeriod = 0
            lastSpeedUpdate = Time.monotonic
          end

          # Copy from the buffer to the output file, up to pos bytes.
          file.write(buf[0...pos])
          bytesLastPeriod += pos

          # Print the progress
          if len > 0
            totalRead += pos
            puts "Downloaded: #{totalRead} / #{len} #{avg.humanize_bytes}/s"
          else
            totalRead += pos
            puts "Downloaded: #{totalRead} / ??? #{avg.humanize_bytes}/s"
          end
        end
      end

      # Done downloading
      downloaded = true
    else
      # Don't handle any others.
      abort "Received HTTP status code #{resp.status_code}"
    end
  end
end

This updates the speed once per second.

3 Likes

It’s exactly what I was looking for ! Thanks a lot my friend !

I will change the appearance, but it’s the base I need :smiling_face_with_three_hearts:

1 Like