Hello, actually I’m writing a package manager for Linux with Crystal.
I have just one problem, I don’t understand how to download a file with crystal.
For example if I want to download this archive:https://download.savannah.gnu.org/releases/acl/acl-2.3.1.tar.xz
How I can do ?
I’m assuming you just want to save the file to the file system? If so it would be something like:
require "http"
HTTP::Client.get "https://download.savannah.gnu.org/releases/acl/acl-2.3.1.tar.xz" do |response|
File.open("out.tar.xz", "w") do |file|
IO.copy response.body_io, file
end
end
However, HTTP::Client doesn’t support redirects at the moment, see HTTP::Client: follow redirects · Issue #2721 · crystal-lang/crystal · GitHub, so you will probably have to add in some extra logic to handle that case. I.e. check for response.status.redirection? and if true, look for the location header and make another request before copying the contents to the file.
Hi again, sorry I know it’s a very old post.
So I followed your example, and I tried with your explanation to have the expected result.
require "http"
HTTP::Client.get("https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.13.12.tar.xz") do |response|
if response.status.redirection?
HTTP::Client.get(response.headers["location"]) do |response|
File.open("linux-5.13.12.tar.xz", "w") do |file|
IO.copy response.body_io, file
end
end
else
File.open("linux-5.13.12.tar.xz", "w") do |file|
IO.copy response.body_io, file
end
end
end
I am not very good actually with networking 
I am definitely sure this can be improve a lot, but I think I didn’t understand everything.
If I use this code, is it okay or it’s a bit slow ?
Like that can I handle multiple redirections ?
How can I mesure received and sended data amount ? Because I would like to provide a nice cli output
I haven’t tried this shard but it looks like it can handle redirects and there’s an example to download a file:
rewrite your code to
require "http/client"
def download_with_redirect(download_link : String, save_path : String = File.basename(download_link), redirect_countdown : Int32 = 1)
HTTP::Client.get(download_link) do |response|
if response.status.success?
File.open(save_path, "w") { |file_io| IO.copy response.body_io, file_io }
elsif !response.status.redirection?
raise "error response" # define your error handling here
elsif redirect_countdown > 0 # follow redirect
new_link = response.headers["localtion"]
download_with_redirect(new_link, save_path, redirect_countdown - 1)
else
raise "too many redirection"
end
end
end
download_with_redirect("https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.13.12.tar.xz", "linux-5.13.12.tar.xz", 5) # 5 is number of redirection attempts
and it should handle redirects correctly.
Oh thanks a lot. How can I calculate the received amount data ? How I now how much data I received ?
A simple way is to just use an intermediate buffer, where you read X bytes from the server, then write X bytes to the destination.
Here’s a whole example client that lets you download something over HTTP(S) to a file and handles redirects.
require "http/client"
abort "No URL" unless ARGV.size > 0
abort "No output file" unless ARGV.size == 2
# Don't clobber files.
abort "File exists" if File.exists?(ARGV[1])
url = ARGV[0]
downloaded = false
# Loop until we've downloaded something (or raise an exception and die spectacularly).
until downloaded
# Do a GET to the url
HTTP::Client.get(url) do |resp|
# Check for a redirect
if resp.status.redirection?
# Get a new URL
if location = resp.headers["location"]
url = location
puts "Redirected to #{url}"
else
abort "Got status #{resp.status_code}, but no location was sent"
end
break # Get out of the .get
end
# Check for status 200
if resp.status_code == 200
# We'll use a buffer so that we can print a percentage.
buf = Bytes.new(65536)
totalRead = 0
len = 0
# If the headers have a Content-Length, then we can provide a
# percentage.
if resp.headers["Content-Length"]?
len = resp.headers["Content-Length"].to_i32
end
# Open the output file
File.open(ARGV[1], "wb") do |file|
# Start reading the data from the HTTP client response.
while (pos = resp.body_io.read(buf)) > 0
file.write(buf[0...pos])
# Print the progress
if len > 0
totalRead += pos
puts "Downloaded: #{totalRead} / #{len}"
else
totalRead += pos
puts "Downloaded: #{totalRead} / ???"
end
end
end
downloaded = true
else
# Don't handle any others.
abort "Received HTTP status code #{resp.status_code}"
end
end
end
Wow nice, thanks a lot.
So I guess in this code the pos variable is the amount of downloaded data ?
I would like to print the download data per second
you can use Time.monotonic to measure time duration between each write loop
the rest is just math.
it will be more accurate if you use channel to print the status every 1 seconds, but the code will be more complex.
Thanks but to be honest I prefer to use the standard API
I added this to one of my programs just last night. Taking the code from before…
require "http/client"
abort "No URL" unless ARGV.size > 0
abort "No output file" unless ARGV.size == 2
abort "File exists" if File.exists?(ARGV[1]) # Don't clobber files.
url = ARGV[0]
downloaded = false
# Loop until we've downloaded something (or raise an exception and die spectacularly).
until downloaded
# Do a GET to the url
HTTP::Client.get(url) do |resp|
# Check for a redirect
if resp.status.redirection?
# Get a new URL
url = resp.headers["location"]? || abort "Got status #{resp.status_code}, but no location was sent"
puts "Redirected to #{url}"
break # Get out of the .get
end
lastSpeedUpdate = Time.monotonic
# Used to hold the average speed
avg = 0
# The number of bytes we've read between updates.
bytesLastPeriod = 0
# Check for status 200
if resp.status_code == 200
# We'll use a buffer so that we can print some progress.
buf = Bytes.new(65536)
totalRead = 0
len = 0
# If the headers have a Content-Length, then we can provide a
# percentage.
len = resp.headers["Content-Length"].to_i32 if resp.headers["Content-Length"]?
# Open the output file
File.open(ARGV[1], "wb") do |file|
# Start reading the data from the HTTP client response.
while (pos = resp.body_io.read(buf)) > 0
# Calculate average speed, one per second.
lapsed = Time.monotonic - lastSpeedUpdate
if lapsed.total_seconds >= 1
div = lapsed.total_nanoseconds / 1_000_000_000
avg = (bytesLastPeriod / div).to_i32!
bytesLastPeriod = 0
lastSpeedUpdate = Time.monotonic
end
# Copy from the buffer to the output file, up to pos bytes.
file.write(buf[0...pos])
bytesLastPeriod += pos
# Print the progress
if len > 0
totalRead += pos
puts "Downloaded: #{totalRead} / #{len} #{avg.humanize_bytes}/s"
else
totalRead += pos
puts "Downloaded: #{totalRead} / ??? #{avg.humanize_bytes}/s"
end
end
end
# Done downloading
downloaded = true
else
# Don't handle any others.
abort "Received HTTP status code #{resp.status_code}"
end
end
end
This updates the speed once per second.
It’s exactly what I was looking for ! Thanks a lot my friend !
I will change the appearance, but it’s the base I need 