Parsing YAML/JSON at compile time

Help & Support
1

I’m working on a shard where YAML and/or JSON files need to be converted to an embedded Hash with a modified structure. This should happen at compile-time because that’s when the data needs to be accessed and validated so that if a key is missing, the compiler raises an error. This is to prevent missing data at runtime.

I’ve tried different setups but can’t make it work. The closest I got was with the run command. Here’s a simplified version to illustrate the setup:

# loader.cr
macro load(path)
  {{ run("./parser/yaml", path) }}
end

The YAML parser looks like this:

# parser/yaml.cr
result = {} of Hash(String, Hash(String, String))

Dir.glob("#{path}/**/*.yml", "#{path}/**/*.yaml") do |file|
  YAML.parse(File.read(file)).as_h.each do |key, data|
    result[key.to_s] = flatten_data_structure(data) # => flattens nested hash
  end
end

puts "#{result}"

Then from the app, I’d like to be able to access the data using a macro:

macro look_up(key)
  \{%
    value = FLATTENED_HASH[{{key}}]
    raise "Missing key #{{{key}}}" if value.is_a?(NilLiteral)
    value
  %}
end

I tried assigning the result of the load macro to a constant, but that doesn’t work. I think because the main script is already compiled at the time the result of the parser is pulled in using the run command.

The only thing I can think of now is first parsing the YAML/JSON files and write the result to a crystal file, which I can then require. That should work, but it adds an extra build step which I’d like to avoid.

What are the alternatives?

1 Like
2

How it doesn’t work? It should work. Do you get an error? What’s the error? Does it produce an empty hash?

3

What I’ve tried, perhaps naively, is assigning it like this:

class Loader
  macro load(path)
    FLATTENED_HASH = {{ run("./parser/yaml", path) }}
  end
end

Loader.load("path/to/yaml")

# => Error: can't declare constant dynamically

And this:

class Loader
  macro load(path)
    {% FLATTENED_HASH = run("./parser/yaml", path) %}
  end
end

Loader.load("path/to/yaml")

# => Error: can only assign to variables, not Path

Then I realised load was being called at runtime. So I tried:

class Loader
  {% begin %}
    {% FLATTENED_HASH = run("./parser/yaml", path) %}
  {% end %}
end

# > 3 | 
# > 4 |         FLATTENED_HASH = 
# > 5 |       
#             ^
# Error: unexpected token: EOF

I’m still wrapping my head around all this. I think the last one is what I’m looking for, but I don’t understand why I’m getting an EOF error. If I change the output of /parser/yaml.cr to something static (e.g. puts %({"a" => "b"}), the EOF error remains.

4

That can never be true. Macros are always executed at compile time.

Can you provide the full code? First, the parser/yaml.cr that you provided doesn’t compile.

A simple reduction works fine for me.

# foo.cr
class Loader
  macro load(path)
    FLATTENED_HASH = {{ run "./bar" }}
  end

  macro look_up(key)
    {% value = FLATTENED_HASH[key] %}
    {% raise "Missing key #{key}" if value.is_a?(NilLiteral) %}
    {{value}}
  end
end

Loader.load("path/to/yaml")
puts Loader.look_up("a")
# puts Loader.look_up("b")

# bar.cr
hash = {"a" => 1}
puts "#{hash}"
1 Like
5

As it turns out, the EOF error was caused by something else. I was running guardian.cr with spec_mirror.cr to run specs on file saves. When I ran crystal spec it worked. Then I restarted guardian and it ran green again.

The full code can be found here: https://github.com/wout/rosetta. It’s intended as an internationalisation shard for LuckyFramework, but it will work outside of Lucky too.

Here is the parser:

Anyway, many thanks for helping me out!

UPDATE
I know now why I was getting one of the errors before on the run macro here:

macro load(path)
  {% parser_name = @type.name.split("::").last.underscore %}

  TRANSLATIONS = {{ run("./parser/#{parser_name.id}", path) }}
end

I used parser_name instead of parser_name.id. :man_facepalming: