Does the array method "<<" always need to be Overloaded?


Hi there,

I didn’t realize this forum existed, I kept going to the Google group!

Anyhow my question is whether the << method for arrays needs to be overloaded to accomodate arrays of user-defined structs.

For example in the code:

def tokenize
    tokens = [] of Token
    until @code.empty?
      tokens << next_token # a method which returns instance of, value)
      @code = @code.strip

The line:
tokens << next_token

gives an error, saying that it could not find any overloads for Array of type Token.

So if I understand, I would need to overload << to tell it what to do when asked to append a new item of type Token?


Can you show the exact error? It seems next_token is returning an array of tokens, not just a single token.


Hi Ary,

Here is the full error report:

Error in instantiating 'Tokenizer#tokenize()'
t ="source.txt")).tokenize
in undefined method '<<' for Nil (compile-time type is (Array(Token) | Nil))
      @tokens << next_token
Rerun with --error-trace to show a complete error trace.

mmm, no next_token only returns a single Token (at least it should!). Below is the full code (some minor changes since my first post as I’ve kept working at it to understand what I might be doing wrong)

class Tokenizer
    {:def, /\bdef\b/},
    {:end, /\bend\b/},
    {:identifier, /\b[a-zA-Z]+\b/},
    {:number, /\b[0-9]+\b/},
    {:oparen, /\(/},
    {:cparen, /\)/},

  def initialize(@code : String)

  def tokenize
    @tokens = [] of Token
    until @code.empty?
      @tokens << next_token
      @code = @code.strip # bad approach here. should have "skip_whitespace"

  def next_token
    TOKEN_KINDS.each do |kind, re|
      re = /\A(#{re})/
      if @code =~ re
        value = $1
        @code = @code[value.size..-1]
        return, value)
      raise"Got stuck at #{@code.inspect}")

struct Token
  property kind, value

  def initialize(@kind : Symbol, @value : String)

t ="source.txt")).tokenize


Aaaah. I think the problem is because “next_token” either returns a Token (by returning early) or not (Nil). And the compiler rightfully infers a return type of Token | Nil, which is my problem I think.

Yes I think this is it…right?


Yes, that’s the reason. Check the compiler error:

in undefined method '<<' for Nil (compile-time type is (Array(Token) | Nil))

It means the compiler figured out @tokens can be nil. And that’s because @tokens isn’t initialized in the initialize method.

That said, there’s no need to use @tokens in your code, just use a local variable tokens. (or so it seems, not sure why you want to store that inside an instance variable)


Yes you’re right about using a local var for “tokens”. There’s no need for it to be an instance var.

I changed tokens to:

tokens = [] of Token | Nil and now things work beautifully.

Thanks for your help Ary!