For the past week or so I’ve been experimenting with creating a Redis-like server in Crystal (which I’ve published under the name Agave) because a few things occurred to me:
- Most of the common data types Redis uses are Crystal language primitives
- The Redis protocol is ridiculously simple (even RESP3 isn’t that bad despite supporting far more data types)
- Using a fiber per connection made it incredibly easy to get up and running
- Redis is single-threaded, so it’s the closest thing to an apples-to-apples performance comparison I’d be able to make
I’m not simply reimplementing Redis, though there is significant overlap with it currently. Instead, I’m trying to make something a bit more consumer-friendly that supports some of the same common use cases. Redis treating nearly everything as arrays of strings (because very little of the Redis ecosystem actually supports RESP3) is pretty irritating to work with in practice.
Use cases I’ve tested so far:
- Caching (
GET cache-key/SET cache-key cached-value EX 3600) - Distributed locking (
SET lock-name lock-value EX 30 NX/EXPIRE lock-name 30) - Distributed counters (
INCR counter/INCRBY counter 10)
And defining new commands is simple. Here’s the actual implementation of GET:
require "../commands"
Agave::Commands.define "get" do
check_expired! key
data[key]?
end
The performance is pretty great so far, too, despite the fact that I’ve done almost no optimization. I’ve been checking performance between Redis and Agave when used as a cache, storing the processed HTML from 100,000 posts on dev.to, all keys having expiration timestamps. After running that on a loop for about 2 hours, this is the current/total resource usage:
I don’t believe this performance will be universally applicable, but considering the most common use case I’ve seen for Redis is as a cache I figured it was the best place to start.
Current features
After only about a week (a holiday week at that), due to Crystal’s expressiveness and performance, I was able to cram in all of these features:
- Data types
- Strings
- Integers
- Floats
- Booleans (not sure how useful this one is, tbh, but I’m experimenting)
- Timestamps
- Lists (implemented with
Array) - Hashes
- Sets
- Setting values only if they do/don’t exist, to support distributed locks
- Key expiration for both caching and crash recovery for holders of distributed locks
- Atomic counters
- I originally implemented integer types with
Atomic(Int64)for lock-freeINCR(fun fact:Atomicdidn’t make values any heavier in memory) but I needed creation and deletion needed to be part of that operation, so I stuck with mutexes
- I originally implemented integer types with
- Automatic snapshots
- Same frequency as Redis in its highest-traffic state
- No AOF equivalent yet
Still needs tests for a lot of functionality, though. Since this started as a throwaway experiment, I’ve written very few. But now that I see performance is so close to Redis even without optimization and it’s been so easy to add features, I’d like to continue development.
