Recursive use of YAML::Builder

Help & Support
1

Bit of a continuation of: How to go about creating a YAML transformer - #4 by svteb

I am attempting to use YAML::Builder to construct a yaml string dynamically based on some nested keys provided in arrays of variable length, i.e. key_path = [:a, :b, ..., :y, :z] where the value for :z key is known:

# Transform using YAML::Builder and serialize to string
def transform : String
  @new_config = YAML.build do |yaml|
    yaml.mapping do
      # Set the config version at the top of the new config
      yaml.scalar("config_version", "v#{ConfigVersion::Latest.value}")

      # Apply transformation rules to the rest of the config
      @transformation_rules.each do |rule|
        value = @old_config.traverse_param(rule[:old_key])
        if value.nil?
          next
        end

        raise "Invalid type #{value.class}" unless value.is_a?(String | Array(String))

        key_path = rule[:new_key]
        apply_rule(value, key_path, yaml)
      end
    end
  end
end

private def apply_rule(value : String | Array(String), key_path : Array(Symbol), yaml : YAML::Builder)
  if key_path.size == 1
    if value.is_a?(Array(String))
      yaml.sequence(key_path.first.to_s) do
        value.each do |item|
          yaml.scalar(item)
        end
      end
    else
      yaml.scalar(key_path.first.to_s, value)
    end
  else
    yaml.mapping(key_path.first.to_s) do 
      apply_rule(value, key_path[1..], yaml)  # Recursive call
    end
  end
end

This currently throws an exception:

./cnf-testsuite transform_config sample-cnfs/sample-coredns-cnf/cnf-testsuite.yml ./TEST.yml
/home/ubuntu/dependencies/crystal-1.6.2-1/share/crystal/src/yaml/builder.cr:209:7 in 'yaml_emit'
/home/ubuntu/dependencies/crystal-1.6.2-1/share/crystal/src/yaml/builder.cr:126:5 in 'end_mapping'
/home/ubuntu/dependencies/crystal-1.6.2-1/share/crystal/src/yaml/builder.cr:133:17 in 'apply_rule'
src/tasks/utils/cnf_installation/config_transformer.cr:122:13 in 'apply_rule'
src/tasks/utils/cnf_installation/config_transformer.cr:103:15 in 'transform'
src/tasks/transform_config.cr:31:5 in '->'
lib/sam/src/sam/task.cr:54:39 in 'call'
lib/sam/src/sam/execution.cr:20:7 in 'invoke'
lib/sam/src/sam.cr:35:5 in 'invoke'
lib/sam/src/sam.cr:53:7 in 'process_tasks'
src/cnf-testsuite.cr:132:3 in '__crystal_main'
/home/ubuntu/dependencies/crystal-1.6.2-1/share/crystal/src/crystal/main.cr:115:5 in 'main_user_code'
/home/ubuntu/dependencies/crystal-1.6.2-1/share/crystal/src/crystal/main.cr:101:7 in 'main'
/home/ubuntu/dependencies/crystal-1.6.2-1/share/crystal/src/crystal/main.cr:127:3 in 'main'
/lib/x86_64-linux-gnu/libc.so.6 in '??'
/lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
./cnf-testsuite in '_start'
???
Error emitting mapping_end

The issue is that I am not exactly sure if I can use this recursive approach to add mappings. That is if I can recursively add a nesting level for each element of some key_path array. I can scantly see that reusing the same reference to yaml in recursive calls could lead to yaml not knowing how to close the mappings when backtracking from the recursion, I’ve tried to create a temporary object that would prevent this:

tmp_yaml = yaml

tmp_yaml.mapping(key_path.first.to_s) do 
   apply_rule(value, key_path[1..], tmp_yaml)  # Recursive call
end

yaml = tmp_yaml

But I still get the same exception.

I should also mention that I added some puts for debugging and the beggining of nesting seems to be going okay:

./cnf-testsuite transform_config sample-cnfs/sample-coredns-cnf/cnf-testsuite.yml ./TEST.yml
Applying rule for key_path: [:deployments, :helm_charts, :name]
Starting mapping for deployments
Applying rule for key_path: [:helm_charts, :name]
Starting mapping for helm_charts
Applying rule for key_path: [:name]
Adding scalar: name => coredns
2

I guess I’ll respond to myself. The code that I provided was not correct in many places and I used the yaml.scalar and yaml.mapping inappropriately. The proper use in apply_rule would be something like this:

private def apply_rule(value : String | Array(String), key_path : Array(Symbol), yaml : YAML::Builder)
  if key_path.size == 1
    if value.is_a?(Array(String))
      yaml.scalar key_path.first
      yaml.sequence do
        value.each do |item|
          yaml.scalar item
        end
      end
    else
      yaml.scalar key_path.first
      yaml.scalar value
    end
  else
    yaml.scalar key_path.first
    yaml.mapping do 
      apply_rule(value, key_path[1..], yaml)  # Recursive call
    end
  end
end

Regardless this will not function with repeat key chains as it will result in something like this:

---
config_version: v2
deployments:
  helm_charts:
    name: coredns
deployments:
  helm_charts:
    helm_repo_name: stable
deployments:
  helm_charts:
    helm_repo_url: https://cncf.gitlab.io/stable
deployments:
  helm_charts:
    helm_chart_name: stable/coredns

You can probably tell this is not desired behavior. To resolve it, it is likely easiest to drop the recursive approach, the key_chains need to be lexicographically sorted and added in an iterative manner where you store the latest parent nesting level, and simply pop out of nesting using the end_mapping when appropriate.

I’ll add one last nitpick and that is my irritation at the lack of proper documentation and examples around YAML::Builder methods.

3

I don’t suppose you have a runnable example of the input and expected output?

I’m sure a PR would be accepted to improve this aspect of things ;)