In one of its answers claude-opus-5.5-high mentioned String::RawConverter, I had never heard of it and a search in Crystal’s source code returned only 4 results! I was pretty surprised an LLM would have such deep knowledge of Crystal. So I thought it would make a good candidate for a benchmark: from the name alone, it’s not obvious that String::RawConverter is an annotation for JSON::Serializable.
So I spammed arena.ai and openrouter.ai with this prompt:
Give two working examples illustrating the use of
String::RawConverterin Crystal.
(Usepp!to print the values.)
The results are classified into 3 categories:
Both examples compile and run.
String::RawConverter is used as an annotation for JSON::Serializable but at least one example fails to compile.
String::RawConverter is not used as an annotation for JSON::Serializable.
| OpenAI | |||
| gpt-6-astra-max | |
||
| gpt-6-astra-xhigh | |
||
| gpt-6-astra-high | |
||
| gpt-6-astra-medium | |
||
| gpt-6-astra-low | |
||
| gpt-6.1-sol-max | |
||
| gpt-6.1-sol-xhigh | |
||
| gpt-6.1-sol-high | |
||
| gpt-6.1-sol-medium | |
||
| gpt-6.1-sol-low | |
||
| gpt-6-sol-max | |
||
| gpt-6-sol-xhigh | |
||
| gpt-6-sol-high | |
||
| gpt-6-sol-medium | |
||
| gpt-6-sol-low | |
||
| gpt-6-sol | |
||
| gpt-6-luna-max | |
||
| gpt-5.6-sol-medium | |
|
|
| gpt-5.6-terra-medium | |
||
| gpt-5.6-terra-low | |
||
| gpt-5.6-luna-xhigh | |
||
| gpt-5.6-luna-medium | |
||
| gpt-5.6-luna-low | |
||
| Anthropic | |||
| claude-fable-5.1-max | |
||
| claude-fable-5.1-high | |
||
| claude-fable-5 | |
||
| claude-opus-5.5-high | |
||
| claude-opus-5.5-medium | |
||
| claude-opus-5.5-low | |
||
| claude-opus-5-high | |
||
| claude-opus-5 | |
||
| claude-opus-4-8-thinking | |
||
| claude-opus-4-8 | |
||
| claude-haiku-4-5-20251001 | |
||
| gemini-3.1-pro-preview-high | |
||
| gemini-3.1-pro-preview-medium | |
||
| gemini-3.1-pro-preview-low | |
||
| gemini-3.8-flash-high | |
||
| gemini-3.8-flash-medium | |
||
| gemini-3.8-flash-low | |
||
| gemini-3.6-flash | |
||
| gemini-3.5-flash | |
||
| gemini-3.5-flash-lite-high | |
||
| gemini-3.5-flash-lite-medium | |
||
| gemini-3.5-flash-lite-low | |
||
| gemini-3.5-flash-lite-minimal | |
||
| gemini-3.5-flash-lite | |
||
| SpaceXAI | |||
| grok-4.7-xhigh | |
||
| grok-4.7-high | |
||
| grok-4.7-medium | |
||
| grok-4.7-low | |
||
| grok-4.6-xhigh | |
|
|
| grok-4.6-high | |
||
| grok-4.5 | |
||
| Meta | |||
| muse-spark-1.3-max | |
|
|
| muse-spark-1.3-xhigh | |
||
| muse-spark-1.3-contributor-xhigh | |
||
| muse-spark-1.3-contributor-high | |
||
| muse-spark-1.3-contributor-medium | |
||
| Tencent | |||
| hy4-preview-high | |
||
| hy4-preview-low | |
||
| hy4-preview | |
|
|
| hy3 | |
||
| Z.ai | |||
| glm-5.3-max | |
||
| glm-5.3-high | |
||
| glm-5.3-low | |
||
| glm-5.3-flash-max | |
||
| glm-5.3-flash-high | |
||
| glm-5.3-flash-low | |
||
| glm-5.3-flash | |
||
| glm-5.1 | |
||
| Qwen | |||
| qwen3.8-max | |
|
|
| qwen3.8-max-fast | |
||
| qwen3.8-omni-flash | |
||
| qwen3.8-omni-flash-fast | |
||
| qwen3.8-27b | |
||
| qwen3.5-397b-a17b | |
||
| Moonshot AI | |||
| kimi-k3-max | |
||
| kimi-k3-high | |
||
| kimi-k3-low | |
||
| kimi-k3 | |
||
| Thinking Machines Lab | |||
| inkling | |
||
| inkling-medium | |
||
| inkling-small-medium | |
||
| inkling-small | |
|
|
| Upstage AI | |||
| solar-pro4 | |
|
|
| solar-pro4-thinking | |
||
| DeepSeek | |||
| deepseek-v4-pro-0813-max | |
||
| deepseek-v4-pro-0813-high | |
||
| deepseek-v4-pro-0813-low | |
||
| deepseek-v4.1-flash-max | |
||
| Others | |||
| mimo-v2.5-pro | |
||
| minimax-m3 | |
||
| mistral-medium-3.5 | |
||
| nvidia-nemotron-3.5-lightning-30b-a3b-nvfp4 | |
||
| nvidia-nemotron-3-ultra-550b-a55b-high | |
||
| nvidia-nemotron-3-ultra | |
||
| step-5-preview | |
||
| Stealth models on openrouter.ai | |||
| space-bunny-alpha-max | |
|
|
| space-bunny-alpha-xhigh | |
||
| space-bunny-alpha-high | |
||
| space-bunny-alpha-medium | |
||
| space-bunny-alpha-low | |
||
| Stealth models on arena.ai | |||
| mimosa | |
|
|
| champagne | |
|
|
| vierra | |
||
| rotten-apple | |
||
| nomagrep | |
