I had not seen LangArena before, so I popped in and … it’s running the crystal code single-threaded, which may explain why Matmul1T has a great score but Matmul4T 8T and 16T do rather horribly so there is some low hanging fruit for optimization there!
Before:
Matmul1T: OK in 5.598s
Matmul4T: OK in 5.809s
Matmul8T: OK in 5.776s
Matmul16T: OK in 5.763s
After:
Matmul1T: OK in 5.637s
Matmul4T: OK in 1.678s
Matmul8T: OK in 1.523s
Matmul16T: OK in 1.737s
I had not seen LangArena before, so I popped in and … it’s running the crystal code single-threaded, which may explain why Matmul1T has a great score but Matmul4T 8T and 16T do rather horribly so there is some low hanging fruit for optimization there!
Yes, that’s what I thought at first.
However, this is probably a benchmark for the initial state of the language. And if that’s the case, it wouldn’t be a fair comparison to enable experimental features only for Crystal. Enabling experimental features in other languages as well would probably lead to further speed improvements.
My assumption was apparently wrong, as the pull request to enable multithreading was merged.
However, since every language probably has experimental optimization techniques, I personally find it difficult to make a fair comparison if only Crystal used them.
crystal top1 
in Expressiveness metric in my bench: LangArena - Programming Languages Benchmark Comparison
second place scala.
Crystal’s expressiveness score is a strong win and a feat that should not be taken lightly. Thanks again @kostya for your huge work on LangArena.
I believe LangArena is good for all mainstream programming languages, not only for Crystal, because we can see with more precision how each language behaves in multiple use cases, to see which one is a better choice according to our needs.
Naturally it also shows how great Crystal is in many of them. IMHO, its combination of efficiency, performance and productivity is almost unmatched.
5 posts were split to a new topic: LLM-friendliness as a metric: porting 20 languages with an LLM
for all languages measures production build, for crystal this is –release. release known to be not incremental.
I do consider compare Incremental compile but use –release no any means
Comparing --release builds with incremental compilation in Crystal doesn’t make much sense. However, for many other languages—like Go, Java, and TypeScript—incremental compilation does produce a production build, so this is a fair comparison across languages.
Rust: 1.72s incremental production build.
Crystal: 23.78s.
This is the fair cost of Crystal’s single-module compilation.
I think we should look at this from the user’s perspective, no one really cares about incremental compilation in release mode, even rust go is more quickly in this case.
Users want to see the result of incremental compilation when do development.
Development compilation does make sense, but there’s no data for that right now. And as I mentioned, Go, Java, and TypeScript produce only one type of build — exactly the production build. There’s no separate ‘development’ build. Still, it would be interesting to see those numbers as well.
Maybe add a new column for Development Incremental compile is a good idea.
