Alternatives To Snake Case? [ String Splitting, Parsing ]

Help & Support
1

Background
Before I go into the particular error I’m encountering, it’s worth mentioning how I got here: I was trying to make a scripting language in Ruby, because I was originally game developing in that language.

But the syntax began rapidly diverging from Ruby because I’m trying to implement a different syntax besides English, specifically using Francais. This is a whole story onto itself, and more in depth than I have time to go into here.

It is presently split into clemots ( equivalent to keywords or function names in Crystal ), numots ( technically suppose to be quoted strings, but I’m having to use snake case at the moment ), and symbols ( exactly the same as Ruby and Crystal symbols ).

Present Issue
In order to chunk each token, I essentially had to reach a document line by line, then after splitting each line by white space ( what seems to be causing issues in the scripting language, but not interactive mode ) I append each token into an array, and then read each array in a Crystal function. As this is is my first scripting language, I have no idea if this is even the right approach. But I’m presently trying to find alternative forms of terminators in string splitting, because “Hello World” keeps becoming “Hello and then World”.

So I’m trying to replace " " termination with multiple delimiters.

  break_points = [%Q(" ), Q( ), Q( ")]
  tokens       = current_line.split(Regex.delimit(break_points))
  total_tokens = tokens.size.to_i
  token_index  = 0

  ## The actual tokens you'll be parsing.
  parsed_tokens = \[\] of String

Which causes this error:

Error: undefined method ‘delimit’ for Regex.class

Is anyone encountering that issue?

Present format
Presently I’m having to use work arounds by relying on solid_snake_case which isn’t super convenient for long term use.

Note before proposing regex, it’s worth point out this doesn’t necessarily fit my use case and is difficult to understand.

The rest of the scripting language seem to work fine oddly enough. It’s just not structured the way people might be used to, as everything is done in one line.

Yes I’ve tried asking other places, but they kept making my problem worse by “refactoring” into a format that isn’t even understood by myself, whose been at this for over a decade.

2

This is no different than me just being like Regex.some_method_that_doesnt_exist("foo"). Can you expand on why you’re thinking that should work? There just isn’t is a .delimit class method, hence the error.

3

There is no .delimit class method in Crystal (nor Ruby). Did you mean .union?

See:

4

I don’t normally use Regex as it’s inconsistent with the rest of the architecture.

This is what I normally use:

  current_line = read_file[index]

  tokens       = current_line.split(" ")
  total_tokens = tokens.size.to_i
  token_index  = 0

  ## The actual tokens you'll be parsing.
  parsed_tokens = [] of String

So I’m curious if Regex is strictly necessary. As when I tried other ways of splitting strings it didn’t really achieve what I was going for. I’m trying to avoid having to use snake_case format. A string is suppose to be “inside of quotes”.

5

Can you provide a minimal example of the input and expected output you’re wanting? That would make this a lot easier to follow/provide suggestions for I’d say.

6

regex isnt strictly necessary for this. if your delimiters are single chars or short strings, just normalize them first, replace each delimiter with a space using gsub, then do a plain split on space. it keeps the tokenizing logic readable and you wont need a Regex.delimit that doesnt exist in crystal.

7

If you insist on doing it that way, then once you have your array of incorrectly parsed tokens, find the first that starts with ". Form there, find the first that ends with ". Join everything from the first to the last in a single token. Repeat.

It’s not the best way to parse a line into tokens, but everyone learns at their own rhythm.