# Asynchronous HTTP requests

**URL:** <https://forum.crystal-lang.org/t/asynchronous-http-requests/466>\
**Category:** Help & Support\
**Created:** [February 24, 2019, 12:33am UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466 "2019-02-24T00:33:43Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![scott](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/scott/32/948_2.png) [@scott](https://forum.crystal-lang.org/u/scott)\
**Post date:** [February 24, 2019, 12:33am UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/1 "2019-02-24T00:33:43Z")

</div>

So, on the vein of @aarongodin’s comments: If you do

```auto
require "http"
addr = "https://jsonplaceholder.typicode.com/todos/1"
HTTP::Client.get addr do |result|
  if result.status_code == 200
    IO.copy result.body_io, STDOUT
    puts "\n ^^ result from #{addr}"
  else
    puts "request failed with status #{result.status_code.inspect}"
  end
end
puts "after the request"
Fiber.yield
puts "after yielding the fiber"

```

The output is:

```auto
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}
 ^^ result from "https://jsonplaceholder.typicode.com/todos/1"
after the request
after yielding the fiber

```

… no spawn - kind of expected, since MT isn’t happening. However, if you do…

```auto
require "http"
addr = "https://jsonplaceholder.typicode.com/todos/1"
spawn do
  HTTP::Client.get addr do |result|
    if result.status_code == 200
      IO.copy result.body_io, STDOUT
      puts "\n ^^ result from #{addr}"
    else
      puts "request failed with status #{result.status_code.inspect}"
    end
  end
end
puts "after the request"
Fiber.yield
puts "after yielding the fiber"

```

…I would expect the result to be

```auto
after the request
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}
 ^^ result from "https://jsonplaceholder.typicode.com/todos/1"
after yielding the fiber

```

… but instead, it is…

```auto
after the request
after yielding the fiber

```

…is there any way to asynchronously perform an HTTP request?? That seems kinda essential…

---

<div class="post-metadata">

**Author:** ![asterite](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/asterite/32/60_2.png) [@asterite](https://forum.crystal-lang.org/u/asterite)\
**Post date:** [February 24, 2019, 1:56am UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/2 "2019-02-24T01:56:34Z")

</div>

Hi!

You might want to read [this](https://crystal-lang.org/reference/guides/concurrency.html)

When you do `Fiber.yield` the runtime checks if there’s another fiber ready to execute. Probably there’s none because the HTTP client is still waiting to hear from the server and so execution continues on the main fiber, prints “after yielding the fiber” and then finishes.

It’s probably easier if you tell us what you want to do and we can tell you how to solve it.

---

<div class="post-metadata">

**Author:** ![straight-shoota](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/straight-shoota/32/36_2.png) [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)\
**Post date:** [February 24, 2019, 7:31am UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/3 "2019-02-24T07:31:45Z")

</div>

You might want to replace `Fiber.yield` with `sleep` and some timeout (or literally anything to fill the fiber). `Fiber.yield` is not guaranteed to yield to the spawned fiber performing the request and even if it does, it might yield back to the main fiber as soon a it hits some IO it needs to wait for. Then the main fiber finishes and the program exits before the request is completed.

---

<div class="post-metadata">

**Author:** ![Sija](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/sija/32/737_2.png) [@Sija](https://forum.crystal-lang.org/u/Sija)\
**Post date:** [February 24, 2019, 1:48pm UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/4 "2019-02-24T13:48:51Z")

</div>

@scott I’d suggest to take a look at [await\_async](https://github.com/anykeyh/await_async) shard which gives you nice syntactic sugar IIUC useful in such scenarios.

---

<div class="post-metadata">

**Author:** ![asterite](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/asterite/32/60_2.png) [@asterite](https://forum.crystal-lang.org/u/asterite)\
**Post date:** [February 24, 2019, 4:10pm UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/5 "2019-02-24T16:10:07Z")

</div>

> I’d suggest to take a look at [await\_async](https://github.com/anykeyh/await_async)

That’s a really unfortunate name. It has nothing to do with traditional `async`/`await` found in other languages like C# and Javascript. This is just a regular `Future`, `Task` or `Promise`.

---

<div class="post-metadata">

**Author:** ![Sija](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/sija/32/737_2.png) [@Sija](https://forum.crystal-lang.org/u/Sija)\
**Post date:** [February 24, 2019, 4:24pm UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/6 "2019-02-24T16:24:59Z")

</div>

True indeed, @anykeyh might be interested in choosing another name for it then… 🙂

---

<div class="post-metadata">

**Author:** ![anykeyh](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/anykeyh/32/726_2.png) [@anykeyh](https://forum.crystal-lang.org/u/anykeyh)\
**Post date:** [February 24, 2019, 6:16pm UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/7 "2019-02-24T18:16:41Z")

</div>

So much trouble I had by naming this gem this way [while it behave exactly like await/async in Scala](https://github.com/scala/scala-async), or in the once awesome [IcedCoffeeScript](http://maxtaco.github.io/coffee-script/). Two languages somehow way better than Javascript ( trolling intended 😸 ).

But yeah, basically `await_async` gives you some syntaxic sugar to wrap promises. It also defer exception raised in fiber to the fiber hanging for result also, for simplicity reasons.  
It has been built to ship small script-like applications, where architecture doesn’t matter so much. I personnaly use it actively in HTTP scrapings tasks.

If your application turns to be bigger, you may want to implement queuing job with channels. They are standards, easy to write, and it will costs you only few more lines of code.

---

<div class="post-metadata">

**Author:** ![asterite](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/asterite/32/60_2.png) [@asterite](https://forum.crystal-lang.org/u/asterite)\
**Post date:** [February 24, 2019, 6:33pm UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/8 "2019-02-24T18:33:17Z")

</div>

> [while it behave exactly like await/async in Scala](https://github.com/scala/scala-async)

No, it doesn’t. Check the scala example. You are supposed to call `await` inside an `async` block. Then Scala probably rewrites the whole thing to continuations or something.

In your case you have `async`, which means `future`, and `await`, which means “wait for the future to complete and get its result”.

I still think that naming this `async`/`await` is not correct.

---

<div class="post-metadata">

**Author:** ![girng](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/girng/32/127_2.png) [@girng](https://forum.crystal-lang.org/u/girng)\
**Post date:** [February 24, 2019, 7:15pm UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/9 "2019-02-24T19:15:52Z")

</div>

IMO, I view this as an issue. The `Fiber.yield` should yield for that previous spawn, and flow of execution behaving how the OP explained is logically sound. That’s probably what most developers think should happen (unless you know the inner workings of the language).

Replacing `Fiber.yield` with sleep, or a timeout to “fill the fiber” would never really come across a developer’s mind because they believe the `Fiber.yield` is waiting for the _previous_ spawn. This is not illogical. @scott Just curious, is this kinda how you felt as well?

edit: I’ve also seen this issue/question being brought up before AFAIK  
edit2: When I say “a developer’s mind”, I mean a non core developer of the language

---

<div class="post-metadata">

**Author:** ![scott](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/scott/32/948_2.png) [@scott](https://forum.crystal-lang.org/u/scott)\
**Post date:** [February 26, 2019, 1:54am UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/10 "2019-02-26T01:54:10Z")

</div>

Sorry that I haven’t been on in a couple days. This topic came up in the gitter channel, and that is why I brought up the question. I have found a solution. As soon as I saw @Sija’s comment about the `await_async` shard, I remembered Crystal’s built-in `Future` implementation! Using this feature solves the issue IMO:

```crystal
require "http"
addr = "https://jsonplaceholder.typicode.com/todos/1"
promise = future do
  HTTP::Client.get addr do |result|
    if result.status_code == 200
      IO.copy result.body_io, STDOUT
      puts "\n ^^ result from #{addr}"
    else
      puts "request failed with status #{result.status_code.inspect}"
    end
  end
end
puts "after the request"
promise.get
puts "after yielding the fiber"

```

---

<div class="post-metadata">

**Author:** ![aarongodin](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/aarongodin/32/266_2.png) [@aarongodin](https://forum.crystal-lang.org/u/aarongodin)\
**Post date:** [February 26, 2019, 4:09am UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/11 "2019-02-26T04:09:28Z")

</div>

Interesting, I was not aware of crystal’s Future method. I’ll try this out as it looks promising (hehe) for what I’m looking to do.

To respond to others asking about the use case for this, I’m a programmer coming from primarily JS where it’s a concern to perform synchronous tasks that halt code execution. I suppose I’m carrying over that concept and wanted to better understand what is happening when I call HTTP::Client.get.

---

<div class="post-metadata">

**Author:** ![stronny](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/stronny/32/228_2.png) [@stronny](https://forum.crystal-lang.org/u/stronny)\
**Post date:** [February 26, 2019, 2:35pm UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/12 "2019-02-26T14:35:16Z")

</div>

If only that was so simple.

Suppose you have a dns name [example.com](http://example.com) with three A records. Suppose that it goes down for some reason. Your call will block for 6 minutes (120s is the default TCP timeout, times the number of IP addresses that the socket wil try to connect to). Finally suppose that you program emits an HTTP request every second to ping something there and I hope you can see a problem.

---

<div class="post-metadata">

**Author:** ![straight-shoota](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/straight-shoota/32/36_2.png) [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)\
**Post date:** [February 26, 2019, 10:17pm UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/13 "2019-02-26T22:17:24Z")

</div>

Eventually, `HTTP::Client` should be able to try to connect to all possible endpoints concurrently if the first one doesn’t respond immediately (way before any regular timeout). We’re not there yet, but I think that’s the plan. The user shouldn’t have to worry about such details.

---

<div class="post-metadata">

**Author:** ![stronny](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/stronny/32/228_2.png) [@stronny](https://forum.crystal-lang.org/u/stronny)\
**Post date:** [February 27, 2019, 12:55pm UTC](https://forum.crystal-lang.org/t/asynchronous-http-requests/466/14 "2019-02-27T12:55:42Z")

</div>

No, but that’s not the point, just an example. My point is that Futures are not good enough an abstraction to just forget about possible problems, because you will end up with millions futures waiting to resolve if for any reason your code starts to block, and that can be anything from sockets to files to child processes to just plain old race conditions.