# Usability of Code Coverage

**URL:** <https://forum.crystal-lang.org/t/usability-of-code-coverage/7249>\
**Category:** Crystal Contrib\
**Created:** [October 3, 2024, 3:43am UTC](https://forum.crystal-lang.org/t/usability-of-code-coverage/7249 "2024-10-03T03:43:11Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Blacksmoke16](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/blacksmoke16/32/1241_2.png) [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)\
**Post date:** [October 3, 2024, 3:43am UTC](https://forum.crystal-lang.org/t/usability-of-code-coverage/7249/1 "2024-10-03T03:43:11Z")

</div>

As per [Code coverage tool anytime soon? · Issue #1157 · crystal-lang/crystal · GitHub](https://github.com/crystal-lang/crystal/issues/1157#issuecomment-2274006954), usage of [GitHub - SimonKagstrom/kcov: Code coverage tool for compiled programs, Python and Bash which uses debugging information to collect and report data without special compilation options](https://github.com/SimonKagstrom/kcov) is currently considered the way forward when it comes to generating code coverage reports in Crystal. However there are a few rough edges I wanted to bring up for discussion to see if they can be improved. Kinda just thinking out loud a bit too.

I started down this path via [Code coverage tool anytime soon? · Issue #1157 · crystal-lang/crystal · GitHub](https://github.com/crystal-lang/crystal/issues/1157#issuecomment-2381352644), but felt a forum thread was more appropriate moving forward.

For purposes of this ticket, let’s assume we’re working with the following two files:

```crystal
# src/test.cr
class Tester
  def foo
    1
  end

  def bar
    2
  end
end

```

```auto
# spec/test_spec.cr
require "../src/test"
require "spec"

describe Tester do
  it "#bar" do
    Tester.new.bar.should eq 2
  end
end

```

## LLVM Optimizations

Generating a kcov report for `Tester` results in a passing spec but an empty kcov report. From what I can tell `Tester#bar` gets inlined/optimized away/never generated? I was under the impression that by default there are no optimizations so this feels a bit unexpected?

Changing the body of `#bar` to `Random.rand(1..10) + 5` results in:

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/crystal_lang/original/2X/2/222b852fb97538660edcf0189c686ddfe5383110.png)

So seems it’s unable to be optimized away so it works correctly this time.

## Dead Code Elimination

In the previous screenshot, notice how the coverage report is “100%” due to the unused `Tester#foo` method being removed from the binary. This is confirmed via:

```sh
$ crystal tool unreachable spec/test_spec.cr
src/test.cr:2:3 Tester#foo 3 lines

```

The `unreachable` tool is able to output JSON, so might be enough to transform the unreachable report into [Codecov Custom Coverage Format](https://docs.codecov.com/docs/codecov-custom-coverage-format) using `0`s to denote those lines were missed. Tho having a way to do what Rust does would be a lot simpler/usable: [Codegen Options - The rustc book](https://doc.rust-lang.org/rustc/codegen-options/index.html#link-dead-code).

EDIT2: TIL about the `--tallies` option to the `unreachable` command. That in of itself is like a coverage report. Wonder how much this differs from kcov…

EDIT3: Given that tool also has a `--format` option, maybe it would be reasonable to support `--format=codecov` 🤔.

## LLVM Coverage

This one is more so for my own learning. I know that kcov uses DWARF debug information to generate its reports. Would there be any functional difference if [LLVM Code Coverage Mapping Format — LLVM 22.0.0git documentation](https://llvm.org/docs/CoverageMappingFormat.html) was being used instead? Like because its native LLVM it’s able to be more accurate/powerful/etc, or is it more of just a different way to achieve the same result?

EDIT: I guess based on the [Rust Issue](https://github.com/rust-lang/rust/issues/34701), it would allow more kinds of coverage to be emitted no matter the platform the user is on. I’m still in the camp that this with native `crystal spec --coverage` would be 💯, but kcov handles things quite well in the meantime…

---

<div class="post-metadata">

**Author:** ![straight-shoota](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/straight-shoota/32/36_2.png) [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)\
**Post date:** [October 3, 2024, 7:24am UTC](https://forum.crystal-lang.org/t/usability-of-code-coverage/7249/2 "2024-10-03T07:24:34Z")

</div>

> [@Blacksmoke16](#):
>
> EDIT3: Given that tool also has a `--format` option, maybe it would be reasonable to support `--format=codecov` 🤔.

That’s definitely possible.

> [@Blacksmoke16](#):
>
> Tho having a way to do what Rust does would be a lot simpler/usable: [Codegen Options - The rustc book](https://doc.rust-lang.org/rustc/codegen-options/index.html#link-dead-code).

In Rust this seems to be a linker config. So it apparently generates code for everything, and usually just doesn’t link unused code.  
The Crystal compiler however does not even run semantic analysis for unreachable code, let alone codegen. And this wouldn’t even be possible to change. A method can only be typed when it’s called and thus the parameter types are known.

---

<div class="post-metadata">

**Author:** ![bcardiff](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/bcardiff/32/3_2.png) [@bcardiff](https://forum.crystal-lang.org/u/bcardiff)\
**Post date:** [October 3, 2024, 1:04pm UTC](https://forum.crystal-lang.org/t/usability-of-code-coverage/7249/4 "2024-10-03T13:04:44Z")

</div>

Good thing we have the unreachable tool. Sounds like a good direction to report a more accurate coverage.

For additional context, if the unused method has arguments without type restriction the compiler can’t know how is expected to be used.

But if those method do have restrictions in all the arguments

```crystal
class Tester
  def foo(x : String)
     # something
  end
end

```

Then we could emit a call `typeof(Tester.new.foo(uninitialized String))` that will force semantic analysis without calling the method (in case it perform some unwanted side effects).

This is to share some direction that might be useful and might offer more information than not-used, but that means it could fail the whole compilation due to an unused method, leading to commenting method. So I’m not 100% certain is more ergonomic.

---

<div class="post-metadata">

**Author:** ![Blacksmoke16](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/blacksmoke16/32/1241_2.png) [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)\
**Post date:** [October 4, 2024, 12:07am UTC](https://forum.crystal-lang.org/t/usability-of-code-coverage/7249/5 "2024-10-04T00:07:36Z")

</div>

```crystal
struct Parent
  def self.some_method
    "foo"
  end
end

pp Parent.some_method

```

```sh
$ crystal tool unreachable test.cr
test.cr:1:1 Parent#initialize lines

```

Is this expected? I guess it’s technically correct, but 🤷.

---

<div class="post-metadata">

**Author:** ![nobodywasishere](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/nobodywasishere/32/1983_2.png) [@nobodywasishere](https://forum.crystal-lang.org/u/nobodywasishere)\
**Post date:** [October 4, 2024, 12:25am UTC](https://forum.crystal-lang.org/t/usability-of-code-coverage/7249/6 "2024-10-04T00:25:30Z")

</div>

I would consider that to be unreachable yeah

---

<div class="post-metadata">

**Author:** ![Blacksmoke16](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/blacksmoke16/32/1241_2.png) [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)\
**Post date:** [October 4, 2024, 12:28am UTC](https://forum.crystal-lang.org/t/usability-of-code-coverage/7249/7 "2024-10-04T00:28:03Z")

</div>

Interestingly only happens for structs, if it were a class there is no output.