# Question about the crystal syntax highlighter

**URL:** <https://forum.crystal-lang.org/t/question-about-the-crystal-syntax-highlighter/7283>\
**Category:** Help & Support\
**Created:** [October 9, 2024, 4:26pm UTC](https://forum.crystal-lang.org/t/question-about-the-crystal-syntax-highlighter/7283 "2024-10-09T16:26:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ralsina](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/ralsina/32/2139_2.png) [@ralsina](https://forum.crystal-lang.org/u/ralsina)\
**Post date:** [October 9, 2024, 4:26pm UTC](https://forum.crystal-lang.org/t/question-about-the-crystal-syntax-highlighter/7283/1 "2024-10-09T16:26:55Z")

</div>

I wrote [GitHub - ralsina/tartrazine: A Crystal reimplementation of the Pygments/Chroma syntax highlighters](http://github.com/ralsina/tartrazine) to highlight code, and because crystal has its own highlighter, I connected both things.

HOWEVER 🙂 it fails with some files which are valid crystal. In particular it failed with [crycco/src/crycco.cr at main · ralsina/crycco · GitHub](https://github.com/ralsina/crycco/blob/main/src/crycco.cr)

I have reduced the failing input to this:

```crystal
/#{l[""]}/
"\n"

```

It fails trying to parse the `\n`

```auto
> bin/tartrazine -f html crycco.cr
Unhandled exception: unknown token: 'n' (Crystal::SyntaxException)
  from /usr/lib/crystal/compiler/crystal/syntax/lexer.cr:2949:7 in 'raise'
  from /usr/lib/crystal/compiler/crystal/syntax/lexer.cr:2948:5 in 'raise'
  from /usr/lib/crystal/compiler/crystal/syntax/lexer.cr:2941:7 in 'unknown_token'
  from /usr/lib/crystal/compiler/crystal/syntax/lexer.cr:158:14 in 'next_token'
  from /usr/lib/crystal/crystal/syntax_highlighter.cr:87:7 in 'highlight_normal_state'
  from /usr/lib/crystal/crystal/syntax_highlighter.cr:80:11 in 'highlight_normal_state'
  from /usr/lib/crystal/crystal/syntax_highlighter.cr:14:5 in 'highlight'
  from src/lexer.cr:412:7 in 'initialize'
  from src/lexer.cr:406:5 in 'new'
  from src/lexer.cr:423:7 in 'tokenizer'
  from src/lexer.cr:422:5 in 'tokenizer'
  from src/formatters/html.cr:79:19 in 'format_text'
  from src/formatters/html.cr:57:7 in 'format'
  from src/main.cr:112:3 in '__crystal_main'
  from /usr/lib/crystal/crystal/main.cr:118:5 in 'main_user_code'
  from /usr/lib/crystal/crystal/main.cr:104:7 in 'main'
  from /usr/lib/crystal/crystal/main.cr:130:3 in 'main'
  from /usr/lib/libc.so.6 in '??'
  from /usr/lib/libc.so.6 in '__libc_start_main'
  from bin/tartrazine in '_start'
  from ???

```

I can produce a minimal program calling the syntax highlighter for a self-contained test if needed

You can also see it in the interpreter, notice how it doesn’t highlight the bad lines:

![image](https://canada1.discourse-cdn.com/flex036/uploads/crystal_lang/original/2X/7/7dd880001e25dee427aa67ac76cf08f7b8303692.png)

---

<div class="post-metadata">

**Author:** ![straight-shoota](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/straight-shoota/32/36_2.png) [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)\
**Post date:** [October 9, 2024, 4:53pm UTC](https://forum.crystal-lang.org/t/question-about-the-crystal-syntax-highlighter/7283/2 "2024-10-09T16:53:49Z")

</div>

This is quite bizarre. The parser is obviously able to parse it and interpret the code. But when parsed for the purpose of syntax highlighting, it fails… Both use cases should usually have the same result 🤔

---

<div class="post-metadata">

**Author:** ![ralsina](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/ralsina/32/2139_2.png) [@ralsina](https://forum.crystal-lang.org/u/ralsina)\
**Post date:** [October 9, 2024, 4:55pm UTC](https://forum.crystal-lang.org/t/question-about-the-crystal-syntax-highlighter/7283/3 "2024-10-09T16:55:55Z")

</div>

I am guessing the nesting of weird and unusual states (a string inside brackets inside an interpolation in a regex) is making the state of the highlighter break, but I looked and zero idea how to fix it 🤷

---

<div class="post-metadata">

**Author:** ![straight-shoota](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/straight-shoota/32/36_2.png) [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)\
**Post date:** [October 9, 2024, 5:52pm UTC](https://forum.crystal-lang.org/t/question-about-the-crystal-syntax-highlighter/7283/4 "2024-10-09T17:52:00Z")

</div>

Yeah, looks like the syntax highlighter does not properly reset `delimiter_state`.

This diff fixes it, although I’m not sure if that’s the correct solution or breaks something else:

```diff
diff --git i/src/crystal/syntax_highlighter.cr w/src/crystal/syntax_highlighter.cr
index 1d4abcb60..a7794e96a 100644
--- i/src/crystal/syntax_highlighter.cr
+++ w/src/crystal/syntax_highlighter.cr
@@ -84,6 +84,8 @@ abstract class Crystal::SyntaxHighlighter
     space_before = false

     while true
+ previous_delimiter_state = lexer.token.delimiter_state
+
       token = lexer.next_token

       case token.type
@@ -105,6 +107,7 @@ abstract class Crystal::SyntaxHighlighter
           highlight_token token, last_is_def
         else
           highlight_delimiter_state lexer, token
+ token.delimiter_state = previous_delimiter_state
         end
       when .string_array_start?, .symbol_array_start?
         highlight_string_array lexer, token

```

---

<div class="post-metadata">

**Author:** ![straight-shoota](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/straight-shoota/32/36_2.png) [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)\
**Post date:** [October 20, 2024, 7:46pm UTC](https://forum.crystal-lang.org/t/question-about-the-crystal-syntax-highlighter/7283/5 "2024-10-20T19:46:31Z")

</div>

Fix: [Fix `SyntaxHihglighter` delimiter state by straight-shoota · Pull Request #15104 · crystal-lang/crystal · GitHub](https://github.com/crystal-lang/crystal/pull/15104)