# Does the array method "\<\<" always need to be Overloaded?

**URL:** <https://forum.crystal-lang.org/t/does-the-array-method-always-need-to-be-overloaded/365>\
**Category:** Learning Resources\
**Created:** [February 2, 2019, 5:24pm UTC](https://forum.crystal-lang.org/t/does-the-array-method-always-need-to-be-overloaded/365 "2019-02-02T17:24:42Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![cloutiy](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/cloutiy/32/222_2.png) [@cloutiy](https://forum.crystal-lang.org/u/cloutiy)\
**Post date:** [February 2, 2019, 5:24pm UTC](https://forum.crystal-lang.org/t/does-the-array-method-always-need-to-be-overloaded/365/1 "2019-02-02T17:24:42Z")

</div>

Hi there,

I didn’t realize this forum existed, I kept going to the Google group!

Anyhow my question is whether the \<\< method for arrays needs to be overloaded to accomodate arrays of user-defined structs.

For example in the code:

```
def tokenize
    tokens = [] of Token
    until @code.empty?
      tokens << next_token # a method which returns instance of Token.new(kind, value)
      @code = @code.strip
    end
    tokens
 end

```

The line:  
`tokens << next_token`

gives an error, saying that it could not find any overloads for Array of type Token.

So if I understand, I would need to overload \<\< to tell it what to do when asked to append a new item of type Token?

---

<div class="post-metadata">

**Author:** ![asterite](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/asterite/32/60_2.png) [@asterite](https://forum.crystal-lang.org/u/asterite)\
**Post date:** [February 2, 2019, 6:28pm UTC](https://forum.crystal-lang.org/t/does-the-array-method-always-need-to-be-overloaded/365/2 "2019-02-02T18:28:13Z")

</div>

Can you show the exact error? It seems next\_token is returning an array of tokens, not just a single token.

---

<div class="post-metadata">

**Author:** ![cloutiy](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/cloutiy/32/222_2.png) [@cloutiy](https://forum.crystal-lang.org/u/cloutiy)\
**Post date:** [February 2, 2019, 8:38pm UTC](https://forum.crystal-lang.org/t/does-the-array-method-always-need-to-be-overloaded/365/3 "2019-02-02T20:38:46Z")

</div>

Hi Ary,

Here is the full error report:

```
Error in compiler.cr:43: instantiating 'Tokenizer#tokenize()'
t = Tokenizer.new(File.read("source.txt")).tokenize
                                           ^ ~~~~~~~
in compiler.cr:17: undefined method '<<' for Nil (compile-time type is (Array(Token) | Nil))
      @tokens << next_token
              ^~
Rerun with --error-trace to show a complete error trace.

```

mmm, no next\_token only returns a single Token (at least it should!). Below is the full code (some minor changes since my first post as I’ve kept working at it to understand what I might be doing wrong)

```
class Tokenizer
  TOKEN_KINDS = [
    {:def, /\bdef\b/},
    {:end, /\bend\b/},
    {:identifier, /\b[a-zA-Z]+\b/},
    {:number, /\b[0-9]+\b/},
    {:oparen, /\(/},
    {:cparen, /\)/},
  ]

  def initialize(@code : String)
  end

  def tokenize
    @tokens = [] of Token
    until @code.empty?
      @tokens << next_token
      @code = @code.strip # bad approach here. should have "skip_whitespace"
    end
    @tokens
  end

  def next_token
    TOKEN_KINDS.each do |kind, re|
      re = /\A(#{re})/
      if @code =~ re
        value = $1
        @code = @code[value.size..-1]
        return Token.new(kind, value)
      end
      raise Exception.new("Got stuck at #{@code.inspect}")
    end
  end
end

struct Token
  property kind, value

  def initialize(@kind : Symbol, @value : String)
  end
end

t = Tokenizer.new(File.read("source.txt")).tokenize
```

---

<div class="post-metadata">

**Author:** ![cloutiy](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/cloutiy/32/222_2.png) [@cloutiy](https://forum.crystal-lang.org/u/cloutiy)\
**Post date:** [February 2, 2019, 8:45pm UTC](https://forum.crystal-lang.org/t/does-the-array-method-always-need-to-be-overloaded/365/4 "2019-02-02T20:45:49Z")

</div>

Aaaah. I think the problem is because “next\_token” either returns a Token (by returning early) or not (Nil). And the compiler rightfully infers a return type of Token | Nil, which is my problem I think.

Yes I think this is it…right?

---

<div class="post-metadata">

**Author:** ![asterite](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/asterite/32/60_2.png) [@asterite](https://forum.crystal-lang.org/u/asterite)\
**Post date:** [February 2, 2019, 9:29pm UTC](https://forum.crystal-lang.org/t/does-the-array-method-always-need-to-be-overloaded/365/5 "2019-02-02T21:29:27Z")

</div>

Yes, that’s the reason. Check the compiler error:

```auto
in compiler.cr:17: undefined method '<<' for Nil (compile-time type is (Array(Token) | Nil))

```

It means the compiler figured out `@tokens` can be `nil`. And that’s because `@tokens` isn’t initialized in the `initialize` method.

That said, there’s no need to use `@tokens` in your code, just use a local variable `tokens`. (or so it seems, not sure why you want to store that inside an instance variable)

---

<div class="post-metadata">

**Author:** ![cloutiy](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/cloutiy/32/222_2.png) [@cloutiy](https://forum.crystal-lang.org/u/cloutiy)\
**Post date:** [February 2, 2019, 11:55pm UTC](https://forum.crystal-lang.org/t/does-the-array-method-always-need-to-be-overloaded/365/6 "2019-02-02T23:55:48Z")

</div>

Yes you’re right about using a local var for “tokens”. There’s no need for it to be an instance var.

I changed `tokens` to:

`tokens = [] of Token | Nil` and now things work beautifully.

Thanks for your help Ary!