I was debugging our failing CI pipeline and found that crystal spec was the culprit, causing out-of-memory on the CI runner. I’ve currently “fixed” it by adding more memory to the CI runner…
Surprised that crystal spec took more memory than all of crystal build, rspec, and webpack (via rails assets:precompile)! Webpack had previously been an out-of-memory problem in our CI pipeline, but was fixed with the environment variable NODE_OPTIONS="--max-old-space-size=...", which tells the Node runtime to run GC more frequently while doing webpack compilation (thanks to this comment).
Now that I think about it, I suppose crystal spec has to pay the entire memory cost of compilation, plus the memory cost of actually running the tests!
Just curious, are there any quick and easy ways to reduce crystal spec memory consumption? Thanks.
crystal spec effectively runs crystal build (IIRC it doesn’t shell out, it executes the equivalent Crystal code), and the compiler often uses > 1GB of RAM alone. But also crystal spec executes the equivalent of crystal run, but it may not have freed up the memory allocated during compilation before it does that, so your entire test suite runs in a separate process, meaning you have both the compiler memory allocated plus whatever your test suite runs being consumed at the same time.
One way you might be able to reduce the memory usage here is to explicitly turn this into two separate steps, which will completely exit the compiler process before running the specs to free up that memory first.
Making it two separate steps executes successfully, thank you! Fairly small drop in measured memory usage; probably only a fix in marginal cases.
I found a somewhat larger ~5% drop using --no-debug on crystal spec (or build). Max RSS size, measured using /usr/bin/time -v crystal spec for example, and clearing rm -rf ~/.cache/crystal/ before each run to simulate a clean CI build environment:
The fact that the compiler is what’s using all the RAM is a good signal. It could’ve been your specs building up a bunch of global state with things like SQLite in-memory storage, Log::MemoryBackend, instrumentation backends designed for testing (which I discovered at a previous company we had deployed to production), and various other constructs designed to hold onto information rather than fire-and-forget so you can inspect them for testing purposes, but it’s good to know that your specs are lightweight!
One thing I’ve done at previous companies was to build artifacts in their own stage that runs on beefier nodes (compiling the spec artifact after building the app itself should be relatively fast since it may have cached a lot of modules from building the app itself) but then the actual test runs happened in a separate stage on smaller, cheaper nodes. If your CI provider charges based on CPU seconds x machine size, this could save some budget. Maybe not, though, if your test runs are quick enough.
GC.collect should never crash anything. If it did, there would be serious issues.
In fact, it can run practically anytime when new memory is allocated. The decision when to actually collect memory is made by BDWGC. The frequency of collection is a trade-off between memory consumption and runtime. If you collect often, you’ll get minimal memory usage but spend a long time checking references.
How does this change affect execution time?
The garbage collector is very generic and has no information about the usage structure of the program it works on.
When running specs, every example is a somewhat contained unit of execution. Most memory allocated in the example should be expected to be out of scope afterwards, so I suppose it could be a good place to collect memory.
Probably the GC is considering there’s more memory available than there is in the system. You can try limit it with setting env variable setting a reasonable n in GC_MAXIMUM_HEAP_SIZE=nG (or M)
So this is running inside a container. It crashes if I do not run GC.collect if I run GC.collect then it does not crash. It is kind of a weird interaction.
This seems relevant to crystal spec memory usage. It looks like GC.collect does not get run while running specs.
Fairly small drop in measured memory usage; probably only a fix in marginal cases.
