The Crystal Programming Language Forum

Agave: Redis-like server in pure Crystal

For the past week or so I’ve been experimenting with creating a Redis-like server in Crystal (which I’ve published under the name Agave) because a few things occurred to me:

  • Most of the common data types Redis uses are Crystal language primitives
  • The Redis protocol is ridiculously simple (even RESP3 isn’t that bad despite supporting far more data types)
  • Using a fiber per connection made it incredibly easy to get up and running
  • Redis is single-threaded, so it’s the closest thing to an apples-to-apples performance comparison I’d be able to make

I’m not simply reimplementing Redis, though there is significant overlap with it currently. Instead, I’m trying to make something a bit more consumer-friendly that supports some of the same common use cases. Redis treating nearly everything as arrays of strings (because very little of the Redis ecosystem actually supports RESP3) is pretty irritating to work with in practice.

Use cases I’ve tested so far:

  • Caching (GET cache-key / SET cache-key cached-value EX 3600)
  • Distributed locking (SET lock-name lock-value EX 30 NX / EXPIRE lock-name 30)
  • Distributed counters (INCR counter / INCRBY counter 10)

And defining new commands is simple. Here’s the actual implementation of GET:

require "../commands"

Agave::Commands.define "get" do
  check_expired! key

  data[key]?
end

The performance is pretty great so far, too, despite the fact that I’ve done almost no optimization. I’ve been checking performance between Redis and Agave when used as a cache, storing the processed HTML from 100,000 posts on dev.to, all keys having expiration timestamps. After running that on a loop for about 2 hours, this is the current/total resource usage:

I don’t believe this performance will be universally applicable, but considering the most common use case I’ve seen for Redis is as a cache I figured it was the best place to start.

Current features

After only about a week (a holiday week at that), due to Crystal’s expressiveness and performance, I was able to cram in all of these features:

  • Data types
    • Strings
    • Integers
    • Floats
    • Booleans (not sure how useful this one is, tbh, but I’m experimenting)
    • Timestamps
    • Lists (implemented with Array)
    • Hashes
    • Sets
  • Setting values only if they do/don’t exist, to support distributed locks
  • Key expiration for both caching and crash recovery for holders of distributed locks
  • Atomic counters
    • I originally implemented integer types with Atomic(Int64) for lock-free INCR (fun fact: Atomic didn’t make values any heavier in memory) but I needed creation and deletion needed to be part of that operation, so I stuck with mutexes
  • Automatic snapshots
    • Same frequency as Redis in its highest-traffic state
    • No AOF equivalent yet

Still needs tests for a lot of functionality, though. Since this started as a throwaway experiment, I’ve written very few. But now that I see performance is so close to Redis even without optimization and it’s been so easy to add features, I’d like to continue development.

16 Likes

This is cool. I am trying to do something similar but instead of a server protocol it would be an embedded DB. Same idea though – use Crystal data types (just Hash, Set, and Array) and have them persist to disk with an AOF similar to Redis.

It’s still early days, but GitHub - mgomes/wraithdb: Embedded NoSQL database for Crystal lang

1 Like

Nice! I didn’t even think about an embedded version, but that’s probably due to working primarily in distributed systems where stateful workloads are deployed and scaled separately from services that consume them.

I like that you use MessagePack for serializing state to disk. I’d started doing that, too, but then I realized I was already using RESP for data over the wire, so I just used that instead. :joy:

1 Like