For the past few days I’ve been seeing some benchmarks floating around Twitter and the fediverse.
There were some others, but these two were the ones I saw the most.
Saving you some clicks: other than JS, they’re all within a few percentage points of each other. In all cases, the benchmark was a web app that both receives and responds with equivalent JSON payloads. I suppose the idea was to benchmark something more substantial than a “hello world” benchmark. I still don’t think that’s enough work to benchmark, but we’ll get to that in a minute.
My understanding is that C# and .NET are used interchangeably. I’ve never worked in that ecosystem, though, so feel free to check me on that.
I ran some benchmarks of my own to see how well Crystal fares because I’m always interested in the relative performance. Now, since these two folks have effectively shown that Swift, .NET, and Rust all perform pretty closely on this particular benchmark, comparing to one of them also compares to the others.
On my machine, the Rust implementation (copy/pasted from the gist linked in the tweet) handled 97k RPS across 1M total requests:
Summary:
Total: 10.3051 secs
Slowest: 0.1218 secs
Fastest: 0.0000 secs
Average: 0.0003 secs
Requests/sec: 97038.9722
This is great. It’s incredibly fast. This gives a baseline for comparison to the Crystal implementation … which does 98k RPS for the same total number of requests (with -Dpreview_mt — important because the other implementations were also using multiple CPU cores):
Summary:
Total: 10.1531 secs
Slowest: 0.2014 secs
Fastest: 0.0000 secs
Average: 0.0003 secs
Requests/sec: 98492.1149
This means that Crystal is as fast as Rust, .NET, and Swift at parsing and emitting these particular JSON payloads. But this isn’t a realistic workload. Parsing and emitting JSON is part a realistic workload, but to dial up the realism a bit, you should at least talk to some kind of data store.
I augmented the Rust benchmark to make a single Redis query while handling the request. This can simulate persisting the data being sent from the client, sticking the data into a queue to be processed asynchronously, or really anything beyond simply transforming one JSON payload into another — there are more efficient ways to do that than sending an HTTP call.
The code is here.
With only that change between them, here are the results from the Rust implementation:
Summary:
Total: 24.4925 secs
Slowest: 0.0704 secs
Fastest: 0.0001 secs
Average: 0.0006 secs
Requests/sec: 40828.8615
And the Crystal implementation:
Summary:
Total: 16.8755 secs
Slowest: 0.0659 secs
Fastest: 0.0000 secs
Average: 0.0004 secs
Requests/sec: 59257.6096
Crystal was on par with Rust when only parsing/emitting HTTP and JSON, but as soon as I added a single data store query in there, Crystal pulled ahead by almost 50%. Even the latency distribution was similar.
Rust:
Latency distribution:
10% in 0.0004 secs
25% in 0.0005 secs
50% in 0.0006 secs
75% in 0.0007 secs
90% in 0.0008 secs
95% in 0.0009 secs
99% in 0.0013 secs
Crystal:
Latency distribution:
10% in 0.0002 secs
25% in 0.0002 secs
50% in 0.0003 secs
75% in 0.0005 secs
90% in 0.0006 secs
95% in 0.0007 secs
99% in 0.0015 secs
Previous notable performance benchmarks I’ve done between Crystal and Rust:
- This very same Redis client outperformed the Rust Redis crate several years ago, too
- Originally, we thought this might’ve been due to the Rust implementation not having been optimized for pipelined queries, but it holds in this case even for a single query
- One I did for a performance optimization in the
will/crystal-pgshard a few years ago showing that, with that optimization, Crystal and Rust perform equivalently on the CPU in talking to Postgres