Hello Crystal community!
For anyone building distributed systems that may scale to large clusters, I just published the crystal-swim shard (MIT license) which implements the SWIM protocol.
(Scalable Weakly-consistent Infection-style Process Group Membership)
What it provides?
A decentralized cluster membership and failure detection protocol.
It solves a similar problem to consensus algorithms (like Raft or Paxos) by keeping a distributed system synchronized. However, while consensus requires strict agreement over data, SWIM provides eventually consistent awareness of who is alive and who is dead.
What is the main benefit?
When eventual consistency is acceptable, using it allows the network load to stay flat regardless of cluster size.
A 10-node cluster and a 1,000-node cluster cost each node about the same few UDP packets per second.
Pure Crystal implementation
It is made in pure Crystal, has no dependencies and includes all the Lifeguard extensions developed by HashiCorp, bringing the most refined version of it to the Crystal community.
It will be useful on other open source projects (all MIT) I’m creating in Crystal as well, and which I will share here soon.
Are there more members here working on distributed systems where this shard would be useful?
Thanks!