HomeBrew update isn’t available yet. brew bump seems to be failing crystal 1.21.0 · Homebrew/homebrew-core@d1b2a06 · GitHub
In src/ameba/tokenizer.cr:88:15
88 | lexer.next_string_array_token
^----------------------
Error: undefined method 'next_string_array_token' for Crystal::Lexer (compile-time type is Crystal::Lexer+)
@Sija How far are we away from ameba 1.7.0?
I’m seeing a performance degradation on MacOS with M5 processor.
hyperfine -w 3 './monyet --eval-fast' './monyet-1.19.1 --eval-fast'
Benchmark 1: ./monyet --eval-fast
Time (mean ± σ): 4.218 s ± 0.185 s [User: 3.658 s, System: 5.218 s]
Range (min … max): 4.072 s … 4.716 s 10 runs
Benchmark 2: ./monyet-1.19.1 --eval-fast
Time (mean ± σ): 2.705 s ± 0.031 s [User: 2.677 s, System: 0.016 s]
Range (min … max): 2.663 s … 2.761 s 10 runs
Summary
./monyet-1.19.1 --eval-fast ran
1.56 ± 0.07 times faster than ./monyet --eval-fast
@MarioAriasC I’m seeing 1.21 monyet using multithreading when I try it locally. It looks like you are, too — 3.7s user + 5.2s system is a lot higher than 4.2s wall-clock time, which indicates you’re using more than 1 CPU core, not to mention 5.2 vs 0.016 system seconds sounds like a lot of contention. Multithreading is definitely gonna change things up.
I tried a different workload (a simple HTTP server benchmark) and I’m also seeing about 14% reduced HTTP throughput on a Linux server running Intel CPUs. The server is simple:
require "http/server"
HTTP::Server.new { }.listen "0.0.0.0", 3000
I deployed this to a production Kubernetes cluster, both on the same node/VM, ran wrk from a different node/VM, and saw numbers like this consistently:
/ # wrk -c30 -d1m http://crystal-1-20-3:3000/ | grep 'Requests/sec'
Requests/sec: 28721.45
/ # wrk -c30 -d1m http://crystal-1-21-0:3000/ | grep 'Requests/sec'
Requests/sec: 25245.78
Both the wrk client and the Crystal servers are running on nodes that are otherwise not under any load.
Running them locally on macOS results in an even worse comparison (288k vs 235k, or about 22% slower on 1.21), but that’s with localhost latency and a fully saturated CPU core vs cross-node latency and ~66-75% utilization, not using multithreading in either one.
➜ Code wrk http://localhost:3001/ | grep 'Requests/sec'
Requests/sec: 235507.58
➜ Code wrk http://localhost:3000/ | grep 'Requests/sec'
Requests/sec: 288240.69
I ran this on both macOS/aarch64 and Linux/Intel to see if the difference in performance shows up across OSes and CPU architectures and it clearly does. If this were a 2-3% variance that would probably be reasonable considering the fundamental difference in fiber scheduling between them. But 14-22% slower across a single release is a pretty wild swing.
Using the fiber qsort bench from @yxhuvud I reach 32ms (EC) vs 56ms at MT:1 for instance, regardless of the Crystal version (1.19+), so the schedulers are better at handling fibers.
Yet, with the blank HTTP server, I indeed notice a -4% impact on performance on x86_64-linux-gnu (Intel 14700k) with EC versus without MT. Still, nothing close to your 14% to 22% @jgaskins. Maybe it has something to do with ARM64 or kqueue, but I don’t have the hardware to reproduce.
The immediate fallback is to compile with -Dwithout_mt. I’m investigating where the 4% I reproduce went, but I’ll need help from someone on macOS and/or aarch64 to do some profiling, too.
Also, irrealistic benchmarks aren’t very representative. Test your actual applications, so thanks @MarioAriasC, I’ll have a look at your app.
@jgaskins Can you add Crystal 1.21 with -Dwithout_mt to your comparison? The fiber context switch for ARM and ARM64 now always uses a memory barrier on every fiber context switch, even for without MT, and I’d like to know how much impact it has compared to 1.20.
can someone do a detailed writeup on how to do realistic benchmarks for concurrency work in Crystal. Also, any such write-ups for performance, concurrency , deadlocks,
I’ll try it and report back
@ysbaddaden @jgaskins -Dwithout_mt did the trick:
❯ hyperfine -w 3 './monyet --eval-fast' './monyet-1.19.1 --eval-fast'
Benchmark 1: ./monyet --eval-fast
Time (mean ± σ): 2.781 s ± 0.049 s [User: 2.739 s, System: 0.020 s]
Range (min … max): 2.728 s … 2.875 s 10 runs
Benchmark 2: ./monyet-1.19.1 --eval-fast
Time (mean ± σ): 2.819 s ± 0.076 s [User: 2.771 s, System: 0.022 s]
Range (min … max): 2.704 s … 2.958 s 10 runs
Summary
./monyet --eval-fast ran
1.01 ± 0.03 times faster than ./monyet-1.19.1 --eval-fast
Monyet has another execution mode with a VM, @jgaskins is familiar with it
hyperfine -w 3 './monyet --vm-fast' './monyet-1.19.1 --vm-fast'
Benchmark 1: ./monyet --vm-fast
Time (mean ± σ): 1.679 s ± 0.074 s [User: 1.636 s, System: 0.011 s]
Range (min … max): 1.590 s … 1.862 s 10 runs
Benchmark 2: ./monyet-1.19.1 --vm-fast
Time (mean ± σ): 1.886 s ± 0.081 s [User: 1.828 s, System: 0.018 s]
Range (min … max): 1.787 s … 2.048 s 10 runs
Summary
./monyet --vm-fast ran
1.12 ± 0.07 times faster than ./monyet-1.19.1 --vm-fast
Gotcha: the performance regression is related to the GC. See Monitor thread causes performance regression · Issue #17144 · crystal-lang/crystal · GitHub for details.
@MarioAriasC Monyet is highly dependent on the GC. A simple top -H -d 0.1 -p <pid> shows that the GC threads are constantly running. Specifying GC_INITIAL_HEAP_SIZE=10M not only drastically improves performance, but the new runtime of Crystal 1.21.0 is faster than the legacy -Dwithout_mt:
$ ./bin/monyet-1.21.0-without_mt --eval-fast
"engine=eval, 9227465, duration=00:00:06.511573880"
$ GC_INITIAL_HEAP_SIZE=10M ./bin/monyet-1.21.0-without_mt --eval-fast
"engine=eval, 9227465, duration=00:00:03.476054929"
$ GC_INITIAL_HEAP_SIZE=10M ./bin/monyet-1.21.0 --eval-fast
"engine=eval, 9227465, duration=00:00:03.004170437"
I’d recommend looking into the GC allocations, there might be lots of allocated objects that are immediately thrown away.
-Dwithout_mt also helped the blank HTTP server on macOS. Running baseline 1.21 on port 3000 and -Dwithout_mt on 3001:
➜ Code wrk http://localhost:3000/ | grep 'Requests/sec'
Requests/sec: 237453.89
➜ Code wrk http://localhost:3001/ | grep 'Requests/sec'
Requests/sec: 286082.57
@ysbaddaden The results I saw on Linux may have been a red herring. I ran them 6-8x and saw consistent results (the 1.21 results were consistently slower, so I showed the fastest of each above) but when I ran them again later after your reply I didn’t see that same consistency, so it may have just been weird luck with latencies between nodes. It’s also surprisingly challenging to synthetically saturate a CPU core over a real network.
Sometimes I had to run wrk -d10s 4 or 5 times in a row to start to get better numbers. It might be prediction branching taking a while to be updated, or the governor allocating more on the E-core rather than P-core, or… 
To help saturate, you’ll want a multiple of 128 connections, because the epoll and kqueue buffers can take up to 128 events, and thus resume 128 fibers then have to wait for wrk to reply to send the next batch (thread sleeps). You can check with tracing enabled, if you only see lone evloop.run blocking=0 (not repeated) and no blocking=1 then the evloop never waits (always something in kernel). Though that will impact latency as events will stay pending in the kernel.
Who said benchmarks are freaking hard? They’re damn right.
What about pre-allocating enough GC HEAP memory to “enough memory” to avoid/reduce GC collections, does it improve anything? Be it macOS on Apple M or Linux on x86_64.
Also, the proposal to use GC_do_blocking is very interesting. It completely fixed the issue for Monyet.
I created a version with a naive implementation of “Frozen Integers”, and I’m seeing better numbers with and without MT (but the difference is bigger with MT)
Going into the rabbit hole, I think that I can switch a big chunk of my classes to structs. I’ll report back.