I have a lot of performance data about tartrazine since it’s a project with a very deterministic output on which I have done extensive performance work, and it’s very easy to benchmark.
I wanted to get a historic perf benchmark for the different releases, which all include static alpine-musl binaries.
The blue lines are benchmarks of each release built on my machine. The red lines are benchmarks of each release using the static binaries from the release (lower is better).
See that performance loss? Static binaries got a lot slower at one point.
Of course this covers many months in time, so it could be many things:
Crystal version
Dependency versions
Alpine version
After lots of compiling and comparing …
In musl, static builds, crystal 1.21.0 is much slower than 1.20.1
In glibc, dynamic builds, 1.21.0 and 1.20.1 perform the same
After debugging, the performance variation can be attributed to … garbage collection.
In 1.21.0 + static + musl the heap size is “grow as you go”
If I set it to start at 16MB then it’s fastest of all compiler versions I tried
1.21.0 static build, sqlite3.h → html
wall time
default GC settings
94.7 ms
GC_INITIAL_HEAP_SIZE=16M
49.2 ms
1.20.1 default (control)
54.5 ms
This is a 50% reduction in wall time, entirely from changing a parameter!!!
Not the dependencies
Not the compiler version
Not the operating system
The heap size.
While this is probably not universally applicable (do you really care about a 40ms startup shave?) it was super fun to figure out!
A bit further investigation, seems like GC initialization is quite a chunk for binaries that do only one thing fast in musl-based systems.
In “normal” dynamic, glibc systems, GC startup is much faster (tens of milliseconds!)
Here are runs on a tiny file where startup (pre-main work) dominates:
config
wall
pre-main (exec→user code)
in-process
Boehm, default
12.7 ms
12.0 ms
0.67 ms
Boehm, GC_INITIAL_HEAP_SIZE=16M
9.4 ms
8.7 ms
0.74 ms
no GC (-Dgc_none)
10.2 ms
9.5 ms
0.68 ms
gcry
12.4 ms
11.7 ms
0.63 ms
And here is a large file where the actual work dominates:
config
wall
pre-main
in-process
Boehm, default
45.4 ms
13.1 ms
32.3 ms
Boehm, GC16
40.0 ms
7.7 ms
32.3 ms
no GC
44.3 ms
10.1 ms
34.2 ms
gcry
44.7 ms
12.8 ms
31.9 ms
I think it’s interesting that starting with a large heap appears to be faster than leaking memory, and also that gcry seems to be within noise of Boehm’s default behaviour!
gcry is %100 Crystal 