Reduce `crystal spec` memory usage?

I was debugging our failing CI pipeline and found that crystal spec was the culprit, causing out-of-memory on the CI runner. I’ve currently “fixed” it by adding more memory to the CI runner… :moneybag:

Surprised that crystal spec took more memory than all of crystal build, rspec, and webpack (via rails assets:precompile)! Webpack had previously been an out-of-memory problem in our CI pipeline, but was fixed with the environment variable NODE_OPTIONS="--max-old-space-size=...", which tells the Node runtime to run GC more frequently while doing webpack compilation (thanks to this comment).

Now that I think about it, I suppose crystal spec has to pay the entire memory cost of compilation, plus the memory cost of actually running the tests!

Just curious, are there any quick and easy ways to reduce crystal spec memory consumption? Thanks.

crystal spec effectively runs crystal build (IIRC it doesn’t shell out, it executes the equivalent Crystal code), and the compiler often uses > 1GB of RAM alone. But also crystal spec executes the equivalent of crystal run, but it may not have freed up the memory allocated during compilation before it does that, so your entire test suite runs in a separate process, meaning you have both the compiler memory allocated plus whatever your test suite runs being consumed at the same time.

One way you might be able to reduce the memory usage here is to explicitly turn this into two separate steps, which will completely exit the compiler process before running the specs to free up that memory first.

TL;DR

In your CI, turn this line:

crystal spec

Into this:

crystal build spec/**/*_spec.cr -o bin/spec
bin/spec
6 Likes

Making it two separate steps executes successfully, thank you! :clap: Fairly small drop in measured memory usage; probably only a fix in marginal cases.

I found a somewhat larger ~5% drop using --no-debug on crystal spec (or build). Max RSS size, measured using /usr/bin/time -v crystal spec for example, and clearing rm -rf ~/.cache/crystal/ before each run to simulate a clean CI build environment:

  • crystal spec 795 MiB
  • crystal build spec/**/*_spec.cr -o out/spec 794 MiB
  • out/spec 20 MiB
  • crystal spec --no-debug 753 MiB
  • crystal build spec/**/*_spec.cr -o out/spec --no-debug 752 MiB

This will buy us a little time but as the code size grows we’ll probably just :moneybag: use the bigger CI instance.

1 Like

The fact that the compiler is what’s using all the RAM is a good signal. It could’ve been your specs building up a bunch of global state with things like SQLite in-memory storage, Log::MemoryBackend, instrumentation backends designed for testing (which I discovered at a previous company we had deployed to production), and various other constructs designed to hold onto information rather than fire-and-forget so you can inspect them for testing purposes, but it’s good to know that your specs are lightweight!

One thing I’ve done at previous companies was to build artifacts in their own stage that runs on beefier nodes (compiling the spec artifact after building the app itself should be relatively fast since it may have cached a lot of modules from building the app itself) but then the actual test runs happened in a separate stage on smaller, cheaper nodes. If your CI provider charges based on CPU seconds x machine size, this could save some budget. Maybe not, though, if your test runs are quick enough. :smile: