Writing tests is an important part of creating an easy to maintain codebase by providing an automated way to ensure your program is doing what it should be. But how do you know if you’re testing the right things, or how effective your tests actually are? Simple: code coverage reporting.
Code coverage reporting is a process in which your specs are ran, and some tool keeps track of what lines of code in your program were executed. From here, the report may then be used to influence where to focus your efforts to improve the coverage percentage, or more ideally ensure all newly added code is covered. But how do you do this with Crystal?
Crystal Code Coverage
Unfortunately there is no super straightforward way to do this via a single --coverage flag when running crystal spec for example. But the overall process is not overly complex, just consists of a few steps:
- Generate the “core” coverage report
- Generate another report representing unreachable methods
- Generate a report for macros
Core Report
The process for this section all was inspired from a blog post by @hanneskaeufler
Given there is no internal way to generate this report within Crystal itself, we need to look for alternatives. The simplest of which is to make use of the fact Crystal uses DWARF for its debug information (the internal data used to power stack traces and such). Knowing this we can use a tool like kcov to read this information to produce our coverage report.
The one problem with kcov however, is that it needs to run against a built binary; meaning we can’t just leverage or tap into crystal spec, but instead must first build a binary that would run the specs when executed. Because there is not single entrypoint into your specs, the easiest way to do this is by creating a file that requires all files within the spec/ directory, then use that as the entrypoint. Something like this, from the root of your shard:
echo 'require "./spec/**"' > './coverage.cr'
mkdir ./bin
crystal build './coverage.cr' -o './bin/coverage'
From here you can run kcov against ./bin/coverage:
kcov --clean --include-path="./src" ./coverage ./bin/coverage --order=random
Let’s break this down:
--cleanmakes it so only the latest run is kept--include-pathwill only includesrc/in the report. I.e. we don’t want code from Crystal’s stdlib or external dependencies to be included./coveragerepresents the directory the report will be written to
The second argument is our built spec binary, which can still accept spec runner options like --order=random.
If all went well you should now have a coverage/index.html file that you can open in your browser to view your core coverage report. It also includes various machine readable coverage report formats that we’ll get to later.
Unreachable Code
Crystal’s compiler removes dead code automatically when building a binary, or in other words, things that are unused (methods, types, etc) will not be included at all in the resulting binary. This is usually a good thing as it’s less code, thus reducing the final binary size.
However, the con of this feature is that because the compiler just totally ignores these unused methods, no type checking occurs on them. This can lead to a sense of false security in that your code could compile just fine, but then start to fail once one of those unused methods starts being used if there is a syntax error within its definition for example. Additionally, kcov is entirely unaware these methods exist and as such do not mark them as missed.
Fortunately for us, there is a built-in tool we can use to identify these unused methods:
crystal tool unreachable --no-color --format=codecov ./coverage.cr > "./coverage/unreachable.codecov.json"
This will output a report marking unreachable methods as missed. More on the --format=codecov in the Tooling section later on.
Macro Code
Up until now, all of the coverage reporting we’ve generated are for the program at runtime. However, Crystal’s macros can be quite complex as well. We can leverage another crystal tool to generate a coverage report for your program’s compile time macro code. This step can be skipped of course if you don’t use any custom macros at all.
crystal tool macro_code_coverage --no-color "./coverage.cr" > "./coverage/macro_coverage.root.codecov.json"
Athena Spec
Athena’s Spec component provides Crystal Spec module compatible testing utilities (NOT a standalone testing framework). One of its particularly useful features is the ability to easy test if code compiles or not, for example:
ASPEC::Methods.assert_compile_time_error "can't instantiate abstract class Foo", <<-CR
abstract class Foo; end
Foo.new
CR
This is especially useful for testing compile time errors as part of custom macro logic. Because of this, it makes use of the same macro_code_coverage tool we used for happy path macro code coverage reporting.
If a ATHENA_SPEC_COVERAGE_OUTPUT_DIR env var is defined and points to an existing directory, the .assert_compile_time_error method will output a macro code coverage report for each invocation of the code it asserted that failed. This ultimately allows the final code coverage report to take these failure flows into account, and not just the happy path.
Tooling
At this point you will have multiple files that each represent a portion of your program’s code coverage. But it’s not super clear how they all fit together. Taking things a step further we can leverage a vendor like Codecov to provide extra capabilities to both make understanding your reports easier, integrate CI checks, and allow sharing results of your project. For example, this is what an Athena PR looks like.
All of the reports we generated are in the codecov custom coverage format. (kcov also generates others which Codecov supports as well). As such, we can upload all of them and Codecov will take care of merging them together into a single view of coverage.
This is as simple as setting up the Codecov Action if you’re using GitHub Actions. For our case, the key thing we need to set is what files to upload, setting the files input to '**/cov.xml,**/unreachable.codecov.json,**/macro_coverage.*.codecov.json' to ensure all the files are uploaded.
There is a lot more nuance to code coverage than what I covered here. The big one being that having 100% test coverage does not imply that your code is bug free, or that it’s even worth trying to get to that level. Instead a good middle ground, for Codecov at least, is to set the target patch percentage to 100% and set project target to auto. These will ensure that all new code is fully covered and does not reduce the overall coverage of the codebase.
Closing Thoughts
What’s next for code coverage reporting in Crystal? My other thread called out some pain points, dead code elimination is essentially solved. But LLVM optimizations can still be a problem that can make it so the reports may not be 100% correct in all scenarios. In my experience this hasn’t affected things all that much however. It would also be interesting to explore an integration with LLVM’s Code Coverage Mapping Format that could possibly make things a bit more accurate.
But I hope this proved useful! Happy to treat this is a living document and keep it updated based on questions or if someone wants me to expand on a certain portion of it. From here I’ll let this sit for a while to gather feedback, if any, then could consider adding some form of this to a sub-page of the Testing page in the Crystal Reference Book.
.
to