I didn’t realize this forum existed, I kept going to the Google group!
Anyhow my question is whether the << method for arrays needs to be overloaded to accomodate arrays of user-defined structs.
For example in the code:
def tokenize
tokens = [] of Token
until @code.empty?
tokens << next_token # a method which returns instance of Token.new(kind, value)
@code = @code.strip
end
tokens
end
The line: tokens << next_token
gives an error, saying that it could not find any overloads for Array of type Token.
So if I understand, I would need to overload << to tell it what to do when asked to append a new item of type Token?
Error in compiler.cr:43: instantiating 'Tokenizer#tokenize()'
t = Tokenizer.new(File.read("source.txt")).tokenize
^~~~~~~~
in compiler.cr:17: undefined method '<<' for Nil (compile-time type is (Array(Token) | Nil))
@tokens << next_token
^~
Rerun with --error-trace to show a complete error trace.
mmm, no next_token only returns a single Token (at least it should!). Below is the full code (some minor changes since my first post as I’ve kept working at it to understand what I might be doing wrong)
class Tokenizer
TOKEN_KINDS = [
{:def, /\bdef\b/},
{:end, /\bend\b/},
{:identifier, /\b[a-zA-Z]+\b/},
{:number, /\b[0-9]+\b/},
{:oparen, /\(/},
{:cparen, /\)/},
]
def initialize(@code : String)
end
def tokenize
@tokens = [] of Token
until @code.empty?
@tokens << next_token
@code = @code.strip # bad approach here. should have "skip_whitespace"
end
@tokens
end
def next_token
TOKEN_KINDS.each do |kind, re|
re = /\A(#{re})/
if @code =~ re
value = $1
@code = @code[value.size..-1]
return Token.new(kind, value)
end
raise Exception.new("Got stuck at #{@code.inspect}")
end
end
end
struct Token
property kind, value
def initialize(@kind : Symbol, @value : String)
end
end
t = Tokenizer.new(File.read("source.txt")).tokenize
Aaaah. I think the problem is because “next_token” either returns a Token (by returning early) or not (Nil). And the compiler rightfully infers a return type of Token | Nil, which is my problem I think.
in compiler.cr:17: undefined method '<<' for Nil (compile-time type is (Array(Token) | Nil))
It means the compiler figured out @tokens can be nil. And that’s because @tokens isn’t initialized in the initialize method.
That said, there’s no need to use @tokens in your code, just use a local variable tokens. (or so it seems, not sure why you want to store that inside an instance variable)