# \`String::RawConverter\` as a nano-benchmark for LLMs

**URL:** <https://forum.crystal-lang.org/t/string-rawconverter-as-a-nano-benchmark-for-llms/9172>\
**Category:** Community\
**Created:** [September 26, 2026, 11:28pm UTC](https://forum.crystal-lang.org/t/string-rawconverter-as-a-nano-benchmark-for-llms/9172 "2026-09-26T23:28:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cristian](https://avatars.discourse-cdn.com/v4/letter/c/a698b9/32.png) [@Cristian](https://forum.crystal-lang.org/u/Cristian)\
**Post date:** [September 26, 2026, 11:28pm UTC](https://forum.crystal-lang.org/t/string-rawconverter-as-a-nano-benchmark-for-llms/9172/1 "2026-09-26T23:28:16Z")

</div>

In one of its answers `claude-opus-5.5-high` mentioned `String::RawConverter`, I had never heard of it and a search in Crystal’s source code returned only [4 results](https://github.com/search?q=repo%3Acrystal-lang%2Fcrystal+RawConverter&type=code)! I was pretty surprised an LLM would have such deep knowledge of Crystal. So I thought it would make a good candidate for a benchmark: from the name alone, it’s not obvious that `String::RawConverter` is an annotation for `JSON::Serializable`.

So I spammed arena.ai and openrouter.ai with this prompt:

> Give two working examples illustrating the use of `String::RawConverter` in Crystal.  
> (Use `pp!` to print the values.)

The results are classified into 3 categories:  
✅ Both examples compile and run.  
⚠ `String::RawConverter` is used as an annotation for `JSON::Serializable` but at least one example fails to compile.  
❌ `String::RawConverter` is not used as an annotation for `JSON::Serializable`.

| | | | |
| --- | --- | --- | --- |
| **OpenAI** | | | |
| gpt-6-astra-max | ✅ | | |
| gpt-6-astra-xhigh | ✅ | | |
| gpt-6-astra-high | ✅ | | |
| gpt-6-astra-medium | ✅ | | |
| gpt-6-astra-low | ✅ | | |
| gpt-6.1-sol-max | ✅ | | |
| gpt-6.1-sol-xhigh | ✅ | | |
| gpt-6.1-sol-high | ✅ | | |
| gpt-6.1-sol-medium | ✅ | | |
| gpt-6.1-sol-low | ✅ | | |
| gpt-6-sol-max | | | ❌ |
| gpt-6-sol-xhigh | | | ❌ |
| gpt-6-sol-high | | | ❌ |
| gpt-6-sol-medium | | | ❌ |
| gpt-6-sol-low | | | ❌ |
| gpt-6-sol | | | ❌ |
| gpt-6-luna-max | | | ❌ |
| gpt-5.6-sol-medium | ✅✅ | | ❌ |
| gpt-5.6-terra-medium | | | ❌ |
| gpt-5.6-terra-low | | | ❌ |
| gpt-5.6-luna-xhigh | | | ❌❌ |
| gpt-5.6-luna-medium | | | ❌❌❌ |
| gpt-5.6-luna-low | | | ❌ |
| **Anthropic** | | | |
| claude-fable-5.1-max | ✅✅ | | |
| claude-fable-5.1-high | ✅ | | |
| claude-fable-5 | ✅ | | |
| claude-opus-5.5-high | ✅✅✅ | | |
| claude-opus-5.5-medium | ✅✅✅ | | |
| claude-opus-5.5-low | ✅✅ | | |
| claude-opus-5-high | ✅✅ | | |
| claude-opus-5 | ✅ | | |
| claude-opus-4-8-thinking | ✅ | | |
| claude-opus-4-8 | | ⚠ | |
| claude-haiku-4-5-20251001 | | | ❌❌ |
| **Google** | | | |
| gemini-3.1-pro-preview-high | ✅ | | |
| gemini-3.1-pro-preview-medium | ✅ | | |
| gemini-3.1-pro-preview-low | ✅ | | |
| gemini-3.8-flash-high | ✅✅✅ | | |
| gemini-3.8-flash-medium | ✅ | | |
| gemini-3.8-flash-low | ✅ | | |
| gemini-3.6-flash | ✅ | | |
| gemini-3.5-flash | ✅ | | |
| gemini-3.5-flash-lite-high | | ⚠ | |
| gemini-3.5-flash-lite-medium | | ⚠ | |
| gemini-3.5-flash-lite-low | | ⚠ | |
| gemini-3.5-flash-lite-minimal | | ⚠ | |
| gemini-3.5-flash-lite | | | ❌ |
| **SpaceXAI** | | | |
| grok-4.7-xhigh | ✅ | | |
| grok-4.7-high | ✅ | | |
| grok-4.7-medium | ✅ | | |
| grok-4.7-low | ✅ | | |
| grok-4.6-xhigh | ✅✅ | ⚠ | |
| grok-4.6-high | ✅ | | |
| grok-4.5 | | ⚠ | |
| **Meta** | | | |
| muse-spark-1.3-max | ✅ | ⚠ | |
| muse-spark-1.3-xhigh | ✅✅ | | |
| muse-spark-1.3-contributor-xhigh | ✅ | | |
| muse-spark-1.3-contributor-high | | ⚠ | |
| muse-spark-1.3-contributor-medium | | ⚠ | |
| **Tencent** | | | |
| hy4-preview-high | ✅ | | |
| hy4-preview-low | | ⚠ | |
| hy4-preview | ✅ | ⚠ | |
| hy3 | | | ❌ |
| **Z.ai** | | | |
| glm-5.3-max | | ⚠ | |
| glm-5.3-high | | ⚠ | |
| glm-5.3-low | | | ❌ |
| glm-5.3-flash-max | ✅ | | |
| glm-5.3-flash-high | | ⚠ | |
| glm-5.3-flash-low | ✅ | | |
| glm-5.3-flash | ✅ | | |
| glm-5.1 | ✅ | | |
| **Qwen** | | | |
| qwen3.8-max | ✅✅ | | ❌❌ |
| qwen3.8-max-fast | | ⚠ | |
| qwen3.8-omni-flash | ✅ | | |
| qwen3.8-omni-flash-fast | | | ❌ |
| qwen3.8-27b | | | ❌❌ |
| qwen3.5-397b-a17b | | | ❌ |
| **Moonshot AI** | | | |
| kimi-k3-max | ✅ | | |
| kimi-k3-high | | | ❌ |
| kimi-k3-low | | | ❌ |
| kimi-k3 | | | ❌ |
| **Thinking Machines Lab** | | | |
| inkling | ✅ | | |
| inkling-medium | | | ❌ |
| inkling-small-medium | | | ❌ |
| inkling-small | ✅ | ⚠ | |
| **Upstage AI** | | | |
| solar-pro4 | ✅ | ⚠ | ❌ |
| solar-pro4-thinking | | | ❌ |
| **DeepSeek** | | | |
| deepseek-v4-pro-0813-max | | | ❌ |
| deepseek-v4-pro-0813-high | | | ❌ |
| deepseek-v4-pro-0813-low | | | ❌ |
| deepseek-v4.1-flash-max | | | ❌❌ |
| **Others** | | | |
| mimo-v2.5-pro | | | ❌❌❌❌ |
| minimax-m3 | | | ❌ |
| mistral-medium-3.5 | | | ❌❌ |
| nvidia-nemotron-3.5-lightning-30b-a3b-nvfp4 | | | ❌❌ |
| nvidia-nemotron-3-ultra-550b-a55b-high | | | ❌ |
| nvidia-nemotron-3-ultra | | | ❌❌ |
| step-5-preview | | | ❌ |
| **Stealth models on openrouter.ai** | | | |
| space-bunny-alpha-max | ✅ | | ❌ |
| space-bunny-alpha-xhigh | | | ❌ |
| space-bunny-alpha-high | | | ❌ |
| space-bunny-alpha-medium | | | ❌ |
| space-bunny-alpha-low | | | ❌ |
| **Stealth models on arena.ai** | | | |
| mimosa | ✅✅✅✅✅ | ⚠⚠⚠ | |
| champagne | | ⚠ | ❌❌ |
| vierra | | | ❌ |
| rotten-apple | | | ❌ |
| nomagrep | | | ❌ |

---

<div class="post-metadata">

**Author:** ![renich](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/renich/32/2137_2.png) [@renich](https://forum.crystal-lang.org/u/renich)\
**Post date:** [September 27, 2026, 6:36am UTC](https://forum.crystal-lang.org/t/string-rawconverter-as-a-nano-benchmark-for-llms/9172/2 "2026-09-27T06:36:13Z")

</div>

I’ll choose to fantasize about me contributing a lot to this since I do all my development with Gemini and it’s almost always in Crystal.

Leave me be!

;D

---

<div class="post-metadata">

**Author:** ![Cristian](https://avatars.discourse-cdn.com/v4/letter/c/a698b9/32.png) [@Cristian](https://forum.crystal-lang.org/u/Cristian)\
**Post date:** [September 27, 2026, 11:39am UTC](https://forum.crystal-lang.org/t/string-rawconverter-as-a-nano-benchmark-for-llms/9172/3 "2026-09-27T11:39:29Z")

</div>

Haha. I’ve been using muse-spark-1.3-contributor over the last few weeks, hoping the next version would be trained on Crystal code. The model is practically free (200 million tokens for $2.34).

---

<div class="post-metadata">

**Author:** ![renich](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/renich/32/2137_2.png) [@renich](https://forum.crystal-lang.org/u/renich)\
**Post date:** [September 27, 2026, 4:36pm UTC](https://forum.crystal-lang.org/t/string-rawconverter-as-a-nano-benchmark-for-llms/9172/4 "2026-09-27T16:36:49Z")

</div>

I wonder how you achieved this. I enrolled into the “use my code for training” thing and I still paid like $70 for having it scan and ansible playbook…

---

<div class="post-metadata">

**Author:** ![Cristian](https://avatars.discourse-cdn.com/v4/letter/c/a698b9/32.png) [@Cristian](https://forum.crystal-lang.org/u/Cristian)\
**Post date:** [September 27, 2026, 5:12pm UTC](https://forum.crystal-lang.org/t/string-rawconverter-as-a-nano-benchmark-for-llms/9172/5 "2026-09-27T17:12:22Z")

</div>

I created an account on [dev.meta.ai](https://dev.meta.ai/) and installed their command-line tool: `curl -fsSL https://dev.meta.ai/install.sh | bash`.  
I opened `muse` and selected `muse-spark-1.3-contributor` with the `/model` command.

 ![muse](https://canada1.discourse-cdn.com/flex036/uploads/crystal_lang/original/2X/5/5c0a876eb41a68bd3a362708598567d1e503c0f3.png)

The model is also available via OpenRouter, though I haven’t tested it: [https://openrouter.ai/meta/muse-spark-1.3-contributor](https://openrouter.ai/meta/muse-spark-1.3-contributor)

---

<div class="post-metadata">

**Author:** ![renich](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/renich/32/2137_2.png) [@renich](https://forum.crystal-lang.org/u/renich)\
**Post date:** [September 27, 2026, 5:29pm UTC](https://forum.crystal-lang.org/t/string-rawconverter-as-a-nano-benchmark-for-llms/9172/6 "2026-09-27T17:29:03Z")

</div>

What about your effort/thinking setting?

---

<div class="post-metadata">

**Author:** ![Cristian](https://avatars.discourse-cdn.com/v4/letter/c/a698b9/32.png) [@Cristian](https://forum.crystal-lang.org/u/Cristian)\
**Post date:** [September 27, 2026, 5:48pm UTC](https://forum.crystal-lang.org/t/string-rawconverter-as-a-nano-benchmark-for-llms/9172/7 "2026-09-27T17:48:16Z")

</div>

It is set to `xhigh`.