I was debugging our failing CI pipeline and found that
crystal spec was the culprit, causing out-of-memory on the CI runner. I’ve currently “fixed” it by adding more memory to the CI runner…
crystal spec took more memory than all of
rails assets:precompile)! Webpack had previously been an out-of-memory problem in our CI pipeline, but was fixed with the environment variable
NODE_OPTIONS="--max-old-space-size=...", which tells the Node runtime to run GC more frequently while doing webpack compilation (thanks to this comment).
Now that I think about it, I suppose
crystal spec has to pay the entire memory cost of compilation, plus the memory cost of actually running the tests!
Just curious, are there any quick and easy ways to reduce
crystal spec memory consumption? Thanks.
crystal spec effectively runs
crystal build (IIRC it doesn’t shell out, it executes the equivalent Crystal code), and the compiler often uses > 1GB of RAM alone. But also
crystal spec executes the equivalent of
crystal run, but it may not have freed up the memory allocated during compilation before it does that, so your entire test suite runs in a separate process, meaning you have both the compiler memory allocated plus whatever your test suite runs being consumed at the same time.
One way you might be able to reduce the memory usage here is to explicitly turn this into two separate steps, which will completely exit the compiler process before running the specs to free up that memory first.
In your CI, turn this line:
crystal build spec/**/*_spec.cr -o bin/spec
Making it two separate steps executes successfully, thank you! Fairly small drop in measured memory usage; probably only a fix in marginal cases.
I found a somewhat larger ~5% drop using
crystal spec (or build). Max RSS size, measured using
/usr/bin/time -v crystal spec for example, and clearing
rm -rf ~/.cache/crystal/ before each run to simulate a clean CI build environment:
crystal spec 795 MiB
crystal build spec/**/*_spec.cr -o out/spec 794 MiB
out/spec 20 MiB
crystal spec --no-debug 753 MiB
crystal build spec/**/*_spec.cr -o out/spec --no-debug 752 MiB
This will buy us a little time but as the code size grows we’ll probably just use the bigger CI instance.
The fact that the compiler is what’s using all the RAM is a good signal. It could’ve been your specs building up a bunch of global state with things like SQLite in-memory storage,
Log::MemoryBackend, instrumentation backends designed for testing (which I discovered at a previous company we had deployed to production), and various other constructs designed to hold onto information rather than fire-and-forget so you can inspect them for testing purposes, but it’s good to know that your specs are lightweight!
One thing I’ve done at previous companies was to build artifacts in their own stage that runs on beefier nodes (compiling the spec artifact after building the app itself should be relatively fast since it may have cached a lot of modules from building the app itself) but then the actual test runs happened in a separate stage on smaller, cheaper nodes. If your CI provider charges based on
CPU seconds x machine size, this could save some budget. Maybe not, though, if your test runs are quick enough.