Usability of Code Coverage

Crystal Contrib
1

As per Code coverage tool anytime soon? · Issue #1157 · crystal-lang/crystal · GitHub, usage of GitHub - SimonKagstrom/kcov: Code coverage tool for compiled programs, Python and Bash which uses debugging information to collect and report data without special compilation options is currently considered the way forward when it comes to generating code coverage reports in Crystal. However there are a few rough edges I wanted to bring up for discussion to see if they can be improved. Kinda just thinking out loud a bit too.

I started down this path via Code coverage tool anytime soon? · Issue #1157 · crystal-lang/crystal · GitHub, but felt a forum thread was more appropriate moving forward.

For purposes of this ticket, let’s assume we’re working with the following two files:

# src/test.cr
class Tester
  def foo
    1
  end

  def bar
    2
  end
end

# spec/test_spec.cr
require "../src/test"
require "spec"

describe Tester do
  it "#bar" do
    Tester.new.bar.should eq 2
  end
end

LLVM Optimizations

Generating a kcov report for Tester results in a passing spec but an empty kcov report. From what I can tell Tester#bar gets inlined/optimized away/never generated? I was under the impression that by default there are no optimizations so this feels a bit unexpected?

Changing the body of #bar to Random.rand(1..10) + 5 results in:

image
image2536×311 27.9 KB

So seems it’s unable to be optimized away so it works correctly this time.

Dead Code Elimination

In the previous screenshot, notice how the coverage report is “100%” due to the unused Tester#foo method being removed from the binary. This is confirmed via:

$ crystal tool unreachable spec/test_spec.cr
src/test.cr:2:3 Tester#foo      3 lines

The unreachable tool is able to output JSON, so might be enough to transform the unreachable report into Codecov Custom Coverage Format using 0s to denote those lines were missed. Tho having a way to do what Rust does would be a lot simpler/usable: Codegen Options - The rustc book.

EDIT2: TIL about the --tallies option to the unreachable command. That in of itself is like a coverage report. Wonder how much this differs from kcov…

EDIT3: Given that tool also has a --format option, maybe it would be reasonable to support --format=codecov :thinking:.

LLVM Coverage

This one is more so for my own learning. I know that kcov uses DWARF debug information to generate its reports. Would there be any functional difference if LLVM Code Coverage Mapping Format — LLVM 20.0.0git documentation was being used instead? Like because its native LLVM it’s able to be more accurate/powerful/etc, or is it more of just a different way to achieve the same result?

EDIT: I guess based on the Rust Issue, it would allow more kinds of coverage to be emitted no matter the platform the user is on. I’m still in the camp that this with native crystal spec --coverage would be :100:, but kcov handles things quite well in the meantime…

2

That’s definitely possible.

In Rust this seems to be a linker config. So it apparently generates code for everything, and usually just doesn’t link unused code.
The Crystal compiler however does not even run semantic analysis for unreachable code, let alone codegen. And this wouldn’t even be possible to change. A method can only be typed when it’s called and thus the parameter types are known.

2 Likes
4

Good thing we have the unreachable tool. Sounds like a good direction to report a more accurate coverage.

For additional context, if the unused method has arguments without type restriction the compiler can’t know how is expected to be used.

But if those method do have restrictions in all the arguments

class Tester
  def foo(x : String)
     # something
  end
end

Then we could emit a call typeof(Tester.new.foo(uninitialized String)) that will force semantic analysis without calling the method (in case it perform some unwanted side effects).

This is to share some direction that might be useful and might offer more information than not-used, but that means it could fail the whole compilation due to an unused method, leading to commenting method. So I’m not 100% certain is more ergonomic.

4 Likes
5 
struct Parent
  def self.some_method
    "foo"
  end
end

pp Parent.some_method

$ crystal tool unreachable test.cr
test.cr:1:1     Parent#initialize        lines

Is this expected? I guess it’s technically correct, but :person_shrugging:.

6

I would consider that to be unreachable yeah

7

Interestingly only happens for structs, if it were a class there is no output.

1 Like