Rspec-retry equivalent for `crystal spec`? Retry intermittently failing tests

Help & Support
1

I have a crystal spec test that is very rarely flaky (fails a few percent of the time) but only flaky in CI, can’t reproduce the issue locally even once despite many attempts. It’s testing a library we use internally at Heii On-Call which does some fancy things in Redis, but the test is entirely sequential and I’ve been unable to determine why it fails, only in CI, and only a tiny percentage of the time. :rage:

In Ruby on Rails we use the rspec-retry gem as a workaround for tests that are flaky in CI. Is there something similar for crystal spec? (If not, can anyone point me in any directions for relatively straightforward ways I might implement it?) Thanks!

2

Might be easier to think about just implementing something for that one spec. Something along the lines of:

it "does a thing" do
  max_tries = 3

  max_tries.times do |t|
    result = some_method_to_calculate_result

    break if result == "expected_value"

    if t == max_tries
      fail "failed to do the thing after #{num_tries} attempts"
    end
  end
end

Be simple enough to have a helper method that you can pass a message/retry count to that will manually fail the spec if a block doesn’t return true within a specific amount of tries.

3

Thanks. This applies to a bunch of similar tests on the same module, and there are a bunch of different assertions that seem to be flaky in this case, so I’ve just cooked up a macro like this:

require "spec"

macro with_retries(max_attempts, &block)
  {{max_attempts}}.times do |%attempt|
    begin
      {{ block.body }}
      break
    rescue ex : Spec::AssertionFailed
      if (%attempt + 1) == {{max_attempts}}
        puts "Attempt ##{%attempt + 1} failed. FAIL."
        raise ex
      else
        puts "Attempt ##{%attempt + 1} failed. Retrying..."
      end
    end
  end
end

describe "Flaky Test" do
  it "works 10% of the time" do
    with_retries(5) do
      Random.rand.should be < 0.1 # assertion is true 10% of the time
    end
  end
end

By catching Spec::AssertionFailed, this is pretty close to what I want!

My issue now is that this does not play nicely with the truncate_db tag that I have to clear the Redis state before each example. (Because all of these attempts are within a single around_each run.) But this might point to the value of the with_clean_db helper method you suggested for me here, since I could do with retries(5) { with_clean_db { ... } }. Thanks @Blacksmoke16 :smile:

4

Doesn’t have to be a macro. Can be just a method, in spec/retry_helper.cr for example:

require "spec"

def with_retries(max_attempts : Int32 = 5, &block)
  max_attempts.times do |attempt|
    begin
      return block.call
    rescue ex : Spec::AssertionFailed
      if (attempt + 1) == max_attempts
        puts "Attempt ##{attempt + 1} failed. FAIL."
        raise ex
      else
        puts "Attempt ##{attempt + 1} failed. Retrying..."
      end
    end
  end
end

(required from spec/spec_helper.cr of course)

5

Is there a reason you’re capturing the block just to do block.call instead of just doing a yield?

6

Hmm, only because of this control structure where I want to use the return to break out of the max_attempts.times loop early. But maybe that would also work with return yield?

7

I think you could use break for that instead to get the same effect.

1 Like
8

May be this can be help?