# Limit concurrency for HTTP::Client

**URL:** <https://forum.crystal-lang.org/t/limit-concurrency-for-http-client/2805>\
**Category:** Help & Support\
**Created:** [December 19, 2020, 7:53pm UTC](https://forum.crystal-lang.org/t/limit-concurrency-for-http-client/2805 "2020-12-19T19:53:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![spinscale](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/spinscale/32/63_2.png) [@spinscale](https://forum.crystal-lang.org/u/spinscale)\
**Post date:** [December 19, 2020, 7:53pm UTC](https://forum.crystal-lang.org/t/limit-concurrency-for-http-client/2805/1 "2020-12-19T19:53:57Z")

</div>

Hey,

I’m currently implementing a daily import jobs against several APIs using Crystal (no need to write Java and save some resources, right 🙂)

While I would like to utilize concurrency and sending several requests at the same time, I do not want to overload the API endpoint and cap the number of concurrent requests. What would be a good approach to that?

The naive solution would be to have a list of items to process, shifting `n` elements of that list, send `n` fibers and use a channel to wait until all fibers have returned. That however means everything is dependent on the slowest fiber. Another idea would be to have block like

```crystal
parallelize(10) do |e|
  # HTTP client request here
end

```

This would require some more synchronization. Another idea could be one of the rate limiter shards, but most of them are based on time (which makes sense for rate limiting i.e. with a token bucket).

I suspect I am thinking too complex and there is something easier to do, so happy to get better ideas.

Happy holidays to everyone!

–Alex

---

<div class="post-metadata">

**Author:** ![oprypin](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/oprypin/32/859_2.png) [@oprypin](https://forum.crystal-lang.org/u/oprypin)\
**Post date:** [December 19, 2020, 8:22pm UTC](https://forum.crystal-lang.org/t/limit-concurrency-for-http-client/2805/2 "2020-12-19T20:22:23Z")

</div>

You are describing a fiber pool.  
An implementation is [https://github.com/akitaonrails/fiberpool](https://github.com/akitaonrails/fiberpool)

---

<div class="post-metadata">

**Author:** ![oprypin](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/oprypin/32/859_2.png) [@oprypin](https://forum.crystal-lang.org/u/oprypin)\
**Post date:** [December 19, 2020, 8:37pm UTC](https://forum.crystal-lang.org/t/limit-concurrency-for-http-client/2805/3 "2020-12-19T20:37:20Z")

</div>

Another implementation that I just put together

```crystal
module Indexable(T)
  def fiberpool_map(workers : Int, &func : T -> R) forall T, R
    Array(R).build(self.size) do |result|
      ch = Channel({Int32, T}).new
      workers.times do
        spawn do
          while (work = ch.receive?)
            i, inp = work
            result[i] = func.call(inp)
          end
        end
      end
      self.each_with_index do |x, i|
        ch.send({i, x})
      end
      ch.close
      self.size
    end
  end
end
 
p (1..100).to_a.fiberpool_map(5, &.**(3))

```

(for actual parallelism see also: [threadpool\_map](https://gist.github.com/00953b336673719dba25))

---

<div class="post-metadata">

**Author:** ![lbarasti](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/lbarasti/32/648_2.png) [@lbarasti](https://forum.crystal-lang.org/u/lbarasti)\
**Post date:** [December 19, 2020, 10:50pm UTC](https://forum.crystal-lang.org/t/limit-concurrency-for-http-client/2805/4 "2020-12-19T22:50:07Z")

</div>

Hey Alex, I think you might find my `rate_limiter` ([https://github.com/lbarasti/rate\_limiter](https://github.com/lbarasti/rate_limiter)) useful to keep an eye on the number of API calls rate.

Going through the readme should be enough for you to figure out whether this is relevant to your use case 👍

---

<div class="post-metadata">

**Author:** ![oprypin](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/oprypin/32/859_2.png) [@oprypin](https://forum.crystal-lang.org/u/oprypin)\
**Post date:** [December 20, 2020, 12:34am UTC](https://forum.crystal-lang.org/t/limit-concurrency-for-http-client/2805/5 "2020-12-20T00:34:25Z")

</div>

Oops just realized that my implementation is broken because it doesn’t wait at the end.  
Exercise for the reader ¯\_(ツ)\_/¯ 🤔