Weird compiling behavior using "benchmark"

Help & Support
1

I have this code that works when I’m not running benchmark.

require "big"

def prime?(n)
  return n | 1 == 3 if n < 5 
  return false if n.gcd(6) != 1 
  p = typeof(n).new(5) 
  until p > Math.isqrt(n)
    return false if n.divisible_by?(p) || n.divisible_by?(p+2) 
    p += 6
  end
  true
end

p = 7919                               # < 2**16
puts "Is #{p} prime? #{prime? p}"

p = 982451653                          # < 2**32
puts "Is #{p} prime? #{prime? p}"

p = 4294967311                         # > 2**32
puts "Is #{p} prime? #{prime? p}"

p = 9223372036854775783i64             # largest I64
puts "Is #{p} prime? #{prime? p}"

p = 12345678901234567891u64            # a UInt64
puts "Is #{p} prime? #{prime? p}"

p = "7000768250000000000000".to_big_i  # a BigInt
puts "Is #{p} prime? #{prime? p}"

=======================
Is 7919 prime? true
Is 982451653 prime? true
Is 4294967311 prime? true
Is 9223372036854775783 prime? true
Is 12345678901234567891 prime? true
Is 7000768250000000000000 prime? false

I can benchmark it using benchmark only if I put the code at the end.

p = 7919                               # < 2**16
puts "Is #{p} prime? #{prime? p}"

p = 982451653                          # < 2**32
puts "Is #{p} prime? #{prime? p}"

p = 4294967311                         # first prime > 2**32
puts "Is #{p} prime? #{prime? p}"

p = 9223372036854775783i64             # largest I64
puts "Is #{p} prime? #{prime? p}"

p = 12345678901234567891u64            # a UInt64
puts "Is #{p} prime? #{prime? p}"

p = "7000768250000000000000".to_big_i  # a BigInt
puts "Is #{p} prime? #{prime? p}"

Benchmark.ips do |b|
    print "\np = #{p}"
    b.report("primep?") { prime?(p) }
    puts
end
=====================
Is 7919 prime? true
Is 982451653 prime? true
Is 4294967311 prime? true
Is 9223372036854775783 prime? true
Is 12345678901234567891 prime? true
Is 7000768250000000000000 prime? false

p = 7000768250000000000000
primep?  78.67M ( 12.71ns) (± 2.93%)  0.0B/op  fastest

But when I put the benchmark code at the top like this I get this type of compiler error.

p = 7919                               # < 2**16
puts "Is #{p} prime? #{prime? p}"
Benchmark.ips do |b|
    print "\np = #{p}"
    b.report("primep?") { prime?(p) }
    puts
end
=============
➜  crystal run --release primalitytestnew.cr --error-trace
In primalitytestnew.cr:64:28

 64 | b.report("prime?") { prime?(p) }
                             ^-------
Error: instantiating 'prime?((Int64 | UInt64))'


In primalitytestnew.cr:9:25

 9 | return false if n.gcd(6) != 1 
                           ^
Error: ambiguous call, implicit cast of 6 matches all of Int64, UInt64

So where I’m doing the benchmark (its position in the code) causes this error.
I assume this is a bug?

2

Benchmark::IPS#report captures the block. This makes p a closured variable. Its type is determined to be a union of all the types it can have when the block executes. If you place the capture on top, this includes the type at this lexical position as well as all types that are later assigned.
If you place the capture on bottom, the type is fixed and won’t change because there are no further assignments.

1 Like
3

Yes. The quick solution was to give each number a separate variable name.
I eventually put all the benchmark code into a method to enable reuse of variable names.

I spent my time debugging thinking it was the code based on the error message.
It would have been clearer if the error was something like:

error: p variable multiple values causing type overload

4

But this isn’t an error. It’s working exactly as it’s supposed to and this can be totally valid and expected behaviour.

It’s the user’s task to make sure their closured variables have the right types (and are not accidentally assigned other types to form a union).
It would be great to raise awareness of this, but I have no idea what we could do to improve that.

5

Well actually it was an error, even though it was user (my) cause.
There was an error message generated by the compiler that was very uninformative.
The issue that was causing the error was not what the error message alluded too.
This has to do with language ergonomics. It’s not enough that the software instructions operate as intended, you also have to account for how users can (unknowingly) still create user errors in their applications.

Examples:
It is a standard design practice that the rear wheel break mechanisms are on the right handlebars of bikes. Also, for cars, the stop|gas pedals are placed in expected positions. So if a bike|car maker switched their positions for their products and people started to crash and cause accidents because of user error, that’s still the manufacturers fault (as case law establishes) because even though the product “worked” as designed, the design was in conflict with the universally expected behavior that users came to rely upon when using the product.

If the error message produced by the compiler had stated as I suggested, that overloading of the p value was the “cause” of the problem (as you did) I would have immediately understood what the real issue was and corrected it, as I did after your explanation of the “real problem”.

Copious and clear documentation, examples, and tutorials!

Rust is a hard language to learn well because it’s so different from what people are used to.
But its developers|community know that, and have developed some of the best (and copious) documentation for it compared to most languages, and their forum is very helpful.

Crystal, of course, has nowhere near the level of $$ and people resources to allocate toward all the areas of its ecosystem it needs to create and upgrade, but that doesn’t mean it doesn’t need to create and develop all these other aspects of its ecosystem. The best technology alone hardly ever wins over human beings.

One of the best allocation of resources I would encourage making is to invest ($$ and people) into making world class educational material (docs, videos, examples, etc). The lack of such has been cited over and over as a major reason many (not just) Open Source projects fail.

These are not criticism of Crystal, but generic observations true of any project.

When I worked for NASA the first thing done was to write system specs, so you would have some idea of if you are on track and schedule.

If there was a ‘gotcha’ section in the docs, that could be contributed to like for Wikipedia, I would contribute lots of examples I’ve encountered in using Crystal. I’m sure other would eagerly do so too. But it’s got to be important to the project to provide the means to do so or it won’t get done.

6

This is a bug, please report it. p never changes after it’s captured so it’s type must be the last one used.

7

Actually, nevermind. It’s not clear what’s the code that fails and what’s the code that succeeds.

1 Like
8

@jzakiya Please provide a complete example of the failing code so we can all be on the same page what we’re actually talking about.

1 Like
9

This is a reduced version of the code.

I have multiple named version of prime? and was benchmarking them for different inputs.
When I have the benchmark code after each p value it gives the error.
It works when I run only one benchmark at the end, or give each value a different name before each benchmark section. Here’s reduced code that produces the errors.

require "big"
require "benchmark"

def prime?(n)
  return n | 1 == 3 if n < 5
  return false if n.gcd(6) != 1
  p = typeof(n).new(5)
  until p > Math.isqrt(n)
    return false if n.divisible_by?(p) || n.divisible_by?(p+2)
    p += 6
  end
  true
end

p = 7919
Benchmark.ips do |b|
  print "\np = #{p} of type #{p.class} is #{prime?(p) ? "prime": "not prime"} "
  b.report("prime?")  { prime?  p }
  puts
end

p = 982451653
Benchmark.ips do |b|
  print "\np = #{p} of type #{p.class} is #{prime?(p) ? "prime": "not prime"} "
  b.report("prime?")  { prime?  p }
  puts
end

p = 4294967311
Benchmark.ips do |b|
  print "\np = #{p} of type #{p.class} is #{prime?(p) ? "prime": "not prime"} "
  b.report("prime?")  { prime?  p }
  puts
end

p = 9223372036854775783i64
Benchmark.ips do |b|
  print "\np = #{p} of type #{p.class} is #{prime?(p) ? "prime": "not prime"} "
  b.report("prime?")  { prime?  p }
  puts
end

p = 12345678901234567891u64
Benchmark.ips do |b|
  print "\np = #{p} of type #{p.class} is #{prime?(p) ? "prime": "not prime"} "
  b.report("prime?")  { prime?  p }
  puts
end

p = "7000768250000000000000".to_big_i 
Benchmark.ips do |b|
  print "\np = #{p} of type #{p.class} is #{prime?(p) ? "prime": "not prime"} "
  b.report("prime?")  { prime?  p }
  puts
end

Here’s reduced code of what I eventually used.

require "big"
require "benchmark"

def prime?(n)
  return n | 1 == 3 if n < 5
  return false if n.gcd(6) != 1
  p = typeof(n).new(5)
  isqrtn = Math.isqrt(n)
  until p > isqrtn
    return false if n.divisible_by?(p) || n.divisible_by?(p + 2)
    p += 6
  end
  true
end

def benchtest(n)
  Benchmark.ips do |b|
    print "\np = #{n} of type #{n.class} is #{prime?(n) ? "prime": "not prime"} "
    b.report("prime ") { prime?  n }
    puts
  end
end

benchtest 5

benchtest 17

benchtest 97

benchtest 541

benchtest 7919

benchtest 982451653

benchtest 4294967311

benchtest 9223372036854775783i64

benchtest 12345678901234567891u6