# Building a web crawler

**URL:** <https://forum.crystal-lang.org/t/building-a-web-crawler/4464>\
**Category:** Help & Support\
**Created:** [March 12, 2022, 7:33pm UTC](https://forum.crystal-lang.org/t/building-a-web-crawler/4464 "2022-03-12T19:33:40Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Inexperienced23](https://avatars.discourse-cdn.com/v4/letter/i/e47774/32.png) [@Inexperienced23](https://forum.crystal-lang.org/u/Inexperienced23)\
**Post date:** [March 12, 2022, 7:33pm UTC](https://forum.crystal-lang.org/t/building-a-web-crawler/4464/1 "2022-03-12T19:33:41Z")

</div>

I have been building a web crawler in ruby, but then decided I did not like how resource intense it was, so I switched to crystal. Anyways, I was using httparty and nokogiri for it, and was wondering if there is anything similar in crystal?

---

<div class="post-metadata">

**Author:** ![jgaskins](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jgaskins/32/2449_2.png) [@jgaskins](https://forum.crystal-lang.org/u/jgaskins)\
**Post date:** [March 12, 2022, 8:33pm UTC](https://forum.crystal-lang.org/t/building-a-web-crawler/4464/2 "2022-03-12T20:33:36Z")

</div>

There sure is, and both are in the standard library so you don’t even need to install anything (other than `libxml2` if your system doesn’t already have it installed).

`HTTP::Client` works a lot like `HTTParty`:

```crystal
require "http"
puts HTTP::Client.get("https://crystal-lang.org/").body

```

You can also instantiate the client to avoid having to create a new TCP connection every time, just like you do with `HTTParty`. [Docs](https://crystal-lang.org/api/1.3.2/HTTP/Client.html)

And then for parsing the HTML bodies, you can use `XML.parse_html`:

```crystal
require "http"
require "xml"

html = HTTP::Client.get(url).body
parsed = XML.parse_html(html)
parsed.xpath_nodes("//a").each do |link|
  puts link["href"]?
end

```

---

<div class="post-metadata">

**Author:** ![Blacksmoke16](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/blacksmoke16/32/1241_2.png) [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)\
**Post date:** [March 12, 2022, 8:55pm UTC](https://forum.crystal-lang.org/t/building-a-web-crawler/4464/3 "2022-03-12T20:55:16Z")

</div>

Before you sink a bunch of time into this, checkout [GitHub - watzon/arachnid: Powerful web scraping framework for Crystal](https://github.com/watzon/arachnid) and see if that would suit your needs.

EDIT: NVM, apparently it’s broken…

\cc @watzon

---

<div class="post-metadata">

**Author:** ![Inexperienced23](https://avatars.discourse-cdn.com/v4/letter/i/e47774/32.png) [@Inexperienced23](https://forum.crystal-lang.org/u/Inexperienced23)\
**Post date:** [March 12, 2022, 11:25pm UTC](https://forum.crystal-lang.org/t/building-a-web-crawler/4464/4 "2022-03-12T23:25:47Z")

</div>

Dang 😂, that would’ve made my life easier.

---

<div class="post-metadata">

**Author:** ![Blacksmoke16](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/blacksmoke16/32/1241_2.png) [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)\
**Post date:** [March 13, 2022, 12:32am UTC](https://forum.crystal-lang.org/t/building-a-web-crawler/4464/5 "2022-03-13T00:32:33Z")

</div>

Based on [Incompatible with crystal v1.1.1 · Issue #8 · watzon/arachnid · GitHub](https://github.com/watzon/arachnid/issues/8), just needs some updates since Crystal `1.0`. Could just make a PR to touch those things up and should be good to go.

---

<div class="post-metadata">

**Author:** ![watzon](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/watzon/32/481_2.png) [@watzon](https://forum.crystal-lang.org/u/watzon)\
**Post date:** [March 28, 2023, 7:03pm UTC](https://forum.crystal-lang.org/t/building-a-web-crawler/4464/6 "2023-03-28T19:03:58Z")

</div>

I really need to fix arachnid. It is a pretty good crawler, I just got too ambitious with the Marionette integration and ended up making it a lot harder to work with. I need to take it back to the drawing board and try to simplify the architecture.

---

<div class="post-metadata">

**Author:** ![bcardiff](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/bcardiff/32/3_2.png) [@bcardiff](https://forum.crystal-lang.org/u/bcardiff)\
**Post date:** [March 28, 2023, 10:49pm UTC](https://forum.crystal-lang.org/t/building-a-web-crawler/4464/7 "2023-03-28T22:49:27Z")

</div>

Back when I did [GitHub - bcardiff/tasko.cr](https://github.com/bcardiff/tasko.cr) the main use case was to power a web crawler. Yet that is in a private repo and not in the samples.

---

<div class="post-metadata">

**Author:** ![zw963](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/zw963/32/1623_2.png) [@zw963](https://forum.crystal-lang.org/u/zw963)\
**Post date:** [March 29, 2023, 4:59am UTC](https://forum.crystal-lang.org/t/building-a-web-crawler/4464/8 "2023-03-29T04:59:32Z")

</div>

> **[GitHub - matthewmcgarvey/selenium.cr: Selenium library for Crystal](https://github.com/matthewmcgarvey/selenium.cr)**
>
> Selenium library for Crystal. Contribute to matthewmcgarvey/selenium.cr development by creating an account on GitHub.