Does the array method "<<" always need to be Overloaded?

Hi there,

I didn’t realize this forum existed, I kept going to the Google group!

Anyhow my question is whether the << method for arrays needs to be overloaded to accomodate arrays of user-defined structs.

For example in the code:

def tokenize
    tokens = [] of Token
    until @code.empty?
      tokens << next_token # a method which returns instance of Token.new(kind, value)
      @code = @code.strip
    end
    tokens
 end

The line:
tokens << next_token

gives an error, saying that it could not find any overloads for Array of type Token.

So if I understand, I would need to overload << to tell it what to do when asked to append a new item of type Token?

Can you show the exact error? It seems next_token is returning an array of tokens, not just a single token.

Hi Ary,

Here is the full error report:

Error in compiler.cr:43: instantiating 'Tokenizer#tokenize()'
t = Tokenizer.new(File.read("source.txt")).tokenize
                                           ^~~~~~~~
in compiler.cr:17: undefined method '<<' for Nil (compile-time type is (Array(Token) | Nil))
      @tokens << next_token
              ^~
Rerun with --error-trace to show a complete error trace.

mmm, no next_token only returns a single Token (at least it should!). Below is the full code (some minor changes since my first post as I’ve kept working at it to understand what I might be doing wrong)

class Tokenizer
  TOKEN_KINDS = [
    {:def, /\bdef\b/},
    {:end, /\bend\b/},
    {:identifier, /\b[a-zA-Z]+\b/},
    {:number, /\b[0-9]+\b/},
    {:oparen, /\(/},
    {:cparen, /\)/},
  ]

  def initialize(@code : String)
  end

  def tokenize
    @tokens = [] of Token
    until @code.empty?
      @tokens << next_token
      @code = @code.strip # bad approach here. should have "skip_whitespace"
    end
    @tokens
  end

  def next_token
    TOKEN_KINDS.each do |kind, re|
      re = /\A(#{re})/
      if @code =~ re
        value = $1
        @code = @code[value.size..-1]
        return Token.new(kind, value)
      end
      raise Exception.new("Got stuck at #{@code.inspect}")
    end
  end
end

struct Token
  property kind, value

  def initialize(@kind : Symbol, @value : String)
  end
end

t = Tokenizer.new(File.read("source.txt")).tokenize

Aaaah. I think the problem is because “next_token” either returns a Token (by returning early) or not (Nil). And the compiler rightfully infers a return type of Token | Nil, which is my problem I think.

Yes I think this is it…right?

Yes, that’s the reason. Check the compiler error:

in compiler.cr:17: undefined method '<<' for Nil (compile-time type is (Array(Token) | Nil))

It means the compiler figured out @tokens can be nil. And that’s because @tokens isn’t initialized in the initialize method.

That said, there’s no need to use @tokens in your code, just use a local variable tokens. (or so it seems, not sure why you want to store that inside an instance variable)

Yes you’re right about using a local var for “tokens”. There’s no need for it to be an instance var.

I changed tokens to:

tokens = [] of Token | Nil and now things work beautifully.

Thanks for your help Ary!