The Crystal Programming Language Forum

Flatten nested `struct` iterators leads to infinite loop

I’ve run into an unexpected (for me) issue recently. Consider the following example where a simple struct is used as an Iterator

struct MyIter
  include Iterator(Int32)

  def initialize; @index = 0; end
  def next; @index == 5 ? stop : (@index += 1); end
end

the important point is that it’s a struct and that its current state is represented by a value type @index. Now, the following code

MyIter.new.map do |i|
  MyIter.new.map do |j|
    j
  end
end.flatten.to_a

enters an infinite loop. The reason is how the flatten iterator is implemented: it manages the nested iterators in an array, thus copying the inner iterator. When it access the inner iterator to advance it, it does not actually modify that iterator but a copy of it (something like @generators.last.next, the next method will be called on a copy, not on the iterator stored in @generators). Hence, the inner iterator will remain on its first element forever.

Of course, the problem vanishes if the iterator is a class not a struct or if its state is not represented by a value type.

In hindsight this is totally expected but it was pretty surprising to me (and took me some time to figure out what’s going on, in particular, because many iterators within the stdlib are implemented as struct, too).

So the real question is: what should be done about this? Is this considered a bug? Should the implementation of Flatten be changed? Should it be mentioned in docs (or is it already and I missed the hint?)? Am I the only one who has been surprised by this?

1 Like

I’d say that’s a bug. As you said, there are lots of iterators in stdlib implemented as a struct. That makes them fast (because no allocation is required) and should typically not be an issue because all the state changes happen internally.
It should be easy to fix Flatten iterator to store the iterator back in the array after it has been advanced.

1 Like

Thanks for your comments. I’ve made a PR fixing the issue.