How are "shards" listed?

Help & Support
#1

Hello,

I was wondering how site like https://shards.info/ or https://crystalshards.org/ were proceeding to obtain the list of all the shards.

Are the shards declared somewhere in a centralized database ? It is looking for repos with “shards.yml” and language:crystal" in Github, Gitlab etc. ?

#2

It is looking for repos with “shards.yml” and language:crystal" in Github

Yeah. I don’t think they look at gitlab, but I’m not sure on that. Mainly just github from what I understand. Here’s just one example from crystalshards.xyz crystalshards/app.cr at master · f/crystalshards · GitHub

#3

So If I create an invalid repo with .cr files and shards.yml would It be listed as a shards ?

#4

I’m not sure what you mean by “invalid repo”, but if it helps, I have this shard GitHub - jwoertink/psst: Send encrypted string between ruby and crystal which is also bundled with a ruby gem, and this repo doesn’t show up in any of the searches as a shard. But I also have this repo which is (was) a valid crystal shard that also doesn’t show up in any searches but if you search in github you’ll find it.

#5

shards.info and crystalshards.org basically just search for Crystal repos with a shard.yml on Github (it seems shards.info also has Gitlab). You can see the implementation in the source code:

https://shardbox.org/ on the other hand is based on a structured catalog of shards.
This catalog basically works like dependencies in a big shard.yml file. The shards are downloaded exactly the same way as shards install would do and thus they can be hosted at any location supported by shards.
The catalog itself is smaller than the entirety of shards listed on the website because dependencies (and recursive sub-dependencies) of listed shards are also included in the database.

1 Like
#6

I see something wrong.

I’ve updated * Scry and lsp some days ago, but only https://shards.info/ has updated the version correctly…

#7

Scry doesn’t seem to be listed in the shardbox catalog and lsp has a different repo: GitHub - jemc/crystal-lsp: Crystal implementation of the Language Server Protocol (https://github.com/Microsoft/language-server-protocol/blob/gh-pages/specification.md)

1 Like
#8

This was the information I was looking for, thank you.

I was wondering about the advantage of a centralized catalog vs searching github / gitlab for specific structure.
Do you know if this is Crystal specifics or if other package manager use this mechanism ?

#9

Many package managers maintain a central index where every package is registered. Then you need neither crawl github nor an external lookup catalog because the index provides all the information about available packages.

But in a decentralized package management system, there are only these two options.
I’m not familiar with specific implementations, though. In the end, the catalog-based approach is somewhat similar to a registry, except that it’s only used for lookup and the dependency manager itself doesn’t rely on it.

1 Like