Redis sentinel support

Help & Support
1

We are looking for a way to use the redis sentinel setup for an application written in crystal but could not find any maintained shard for this.

The only implementation we found is GitHub - pgeraghty/redis-sentinel-crystal: Redis Sentinel support for Crystal · GitHub but this is 7 years old and not maintained. We currently use jgaskins/redis for plain redis and like it but it has no sentinel support.

Any pointers for this?

2

In a small Crystal project I’m leading, the way we see Redis is this:

  • we want to postpone the “redis tax” (complexity) of Redis Cluster as maximum as possible
  • we already deployed Redis Cluster with 6 nodes: it is complex to setup, even more complex to manage
  • we know that only with Redis Cluster provides real horizontal scalability for read and writes, including data size scalability
  • Redis Sentinel could be a simpler option in the short term, because while being simpler, it allows at least scaling the reads (no write scaling, no data size scaling)

We tested or analyzed some solutions, like:

  • Twemproxy - mature but limited and non-actively maintained anymore
  • Predixy - powerful, but not mature and only a few maintainers: risky
  • KeyDB - in their Active-Active setup (or Active Replica) setup, it is better than Redis Sentinel because you can load-balance both reads and writes. But, there is no sharding and, thus, no horizontal scalability for data size. All data lives on all nodes, with the entire dataset in RAM

Then we found DragonflyDB

  • as per their benchmark, it provides 25x more performance per node than Redis
  • thus, one node can provide the performance of 25 Redis Nodes
  • it also provides Cluster Mode, identical to Redis Cluster, but the cluster management tool is proprietary (Dragonfly Swarm, as per docs)

Our particular conclusion

  • it is interesting to start with DragonflyDB in single instance and vertically scale it as maximum as possible
  • it has the potential to replace a redis Cluster with 25 nodes with similar hardware profile
  • then, when our project really needs more than that, it is already a commercial success, and we have the financial capacity to decide between a Redis Cluster or a DragonflyDB Cluster with Dragonfly Swarm

YMMV, but this may be useful for you and/or for others.

3

Thanks for these insights!

We are actually using keydb at the moment but it is not maintained anymore unfortunately and the DevOps team wants to get away from it because we even found that active-active does not always work on node crash/recovery - there were some operational issues. The main benefit was/is that we could speak plain redis, no client side code needed to benefit from the setup.

We do have other components in the application based on ruby/rails which need a redis compatible key/value store for multiple use cases (locking, temporary data store, even cache) and found that we can not easily migrate away from it but we want to invest into a solution like Redis Sentinel. Our favourite currently is ValKey which supports the sentinel protocol and setup and would allow us to use redis clients as we use now.

The only missing part currently is crystal-lang support for using Redis Sentinel. That’s why I’m asking around here, we have been thinking of implementing this on top of a solution like GitHub - jgaskins/redis: Pure-Crystal Redis client, supporting clustering, RedisJSON, RediSearch, and RedisTimeSeries · GitHub but are still in the discovery phase for our options.

4

Understood. Since DragonflyDB also provides a fully redis-compatible protocol, I would like to understand your team decisions on why/why not using it.

It doesn’t require the sentinel protocol, and jgaskins/redis should theoretically work on it as a drop-in-replacement.

5

Can confirm. I use it with DragonflyDB at my day job.

@dup2 My Redis shard also works with Redis Sentinel despite not being Sentinel-aware, as long as you’re using Redis::ReplicationClient. I’ve used it with a Sentinel setup in the past and it handles failover despite never talking to the Sentinels. It polls for topology changes periodically, so you’ll never lose write access for longer than topology_ttl after the Sentinels complete a failover. The default is 10 seconds which IIRC is shorter than the default Sentinel failover time.

If the ReplicationClient doesn’t meet your needs (because you need to optimize for consistency over availability and writing redis.on_primary(&.get(key)) is tedious), open an issue and we can collaborate on a SentinelClient.

6

Thanks for these pointers - I’ll try and come back to it.

As for DragonflyDB, it is definitely interesting but there are other people involved in the decision who happen to run ValKey with Sentinel already so they are kind of hard to convince to change this setup and would rather want us to be compatible with this as we move away from the keydb setup. This is an infrastructure decision I do somewhat understand, you do not want to manage two different products for the same purpose.

7

Yeah, operating DragonflyDB is very different from operating Redis, so if you’re already using a Redis Sentinel setup and the team is familiar with it, I wouldn’t push too hard to change it unless you had a solid reason.

Our reason was that we needed shared storage for our rate-limiting proxies so each proxy instance wouldn’t have separate counts of how many requests you’ve made. DragonflyDB has a command for that built in. They also provide a Kubernetes operator that handles replication and failover, so it fit both our very narrow use case for it and our infrastructure and nobody on the team had a better option.

For everything else, we use OG Redis instances for exactly the reasons you mention: it’s working for us, the rest of the team is already comfortable with it, and changing minds can be hard, especially when those minds are attached to loud voices. My time is finite and any time I spend navigating their what-ifs about tech that’s new to them is time I’m not spending on problems with real impact.

8

Just wanted to menton Valkey. It’s not written in crystal. It’s it’s own thing. A Redis fork: https://valkey.io/