I would take the TechEmpower benchmarks with a grain of salt. Even their most realistic benchmarks are pretty far from real-world code.
As with all benchmarks, running your own code is the best indicator of how something will perform for your use case rather than trusting contrived benchmarks. If Crystal is the highest performer by that large a margin and performance is one of the top criteria, then it sounds like Crystal is the best choice for this service.
thanks for your valuable inputs & suggestions
techempower framework benchmark (TFB) has very complete benchmark data
there are things that i found so far, regarding my simple benchmark :
actix based web service implementation
it’s very possibly that i havent found the correct way to get the maximum performance in using actix
so, i think i’ll consult with rust users for the correct best practice in using actix
and/or adding a go-lang framework for comparison, as go-lang seems easier to learn than rust
Sort of, I mean I could switch to Redis and win all the benchmarks.
Kemel and Raze are not using ORMs whereas Amber and SG are.
So it’s kind of an apples to oranges comparison and properly misleading to the casual observer.
There are different types of Techempower’s benchmarks. Some of them are meant to test specifically the typical use of different web frameworks (Classification: Fullstack). E.g. you would use say ECR templates in framework x but plain methods in framework y. There will be many other differences and such benchmarks would measure the default (recommended) way of coding in particular framework.
For example why the Rails performs much worse? Part of it is Ruby of course, but it also the fact that Rails has a long middleware stack doing lots of things. If you recreate such stack doing all the same things in Crystal the Crystal benchmarks can be slowed down. Or you can remove some default handlers from Rails middleware list and improve Ruby results. But that’s probably not what you’ll do in real life. Most likely you’ll chose some framework and follow it’s way.
If you want to beat the benchmarks then Redis is not an option as it is single threaded.
Look at Aerospike (aerospike.com) as it is truly multithreaded and have much better features than Redis.
I had big issues with Redis on one of my big projects (I recommended Aerospike but CTO was stuck with Redis, even though I did explain where and how Redis will break)
here i just want to share some additional info, that might be useful
regarding actix, i’ve asked for suggestions in rust forum, here :
AFAIK,
it’s very possibly for a newbie in rust, to follow a simple example in tutorials in internet,
to write an implementation of web service using actix, like in this article (that can be easily found from the result of googling over these words : “rest api with actix web postgresql”) : https://turreta.com/2019/09/21/rest-api-with-rust-actix-web-and-postgresql-part-1/
but if we read more on actix, which is basically async (which is different from other rust web frameworks which is sync : nickel, iron, rocket, etc.), then we can find that the example in that article is sync or blocking.
in another word, it’s the wrong way of using actix, as the performance will be very low.
one of the best practice in using actix with database access is written in the official doc/example :
blocking parts of code must be put inside web::block(move || { ... })
one of an async implementation that i’ve tried, is more than 10 times faster than the sync version.
regarding the multi core cpu, i’ve made the same benchmark in 4 core cpu (Xeon CPU E3-1225 v5 @ 3.30GHz).
when using 4 concurrent client connections, crystal based frameworks are still relatively fast.
but when the concurrency gets increased, the results are different.
and in 64 concurrent connections, actix diesel is faster :
Out of curiosity, what is the CPU usage of the various processes in this benchmark? Is it possible that Crystal is serving more requests per second per CPU core consumed and that building with -Dpreview_mt would improve its throughput?
yes, about go, i’ve tried some go web framework yesterday (gorilla, fasthttp-router,…), to get some more comparisons.
for rust, to develop web service in rust, it’s very common to use ORM (mostly used: diesel).
so in my previous tests, i used diesel as the db connection layer.
today, i try not to use ORM, i use tokio-postgres (an async library to access postgress)
Could you try doing the same. but with wrk instead of ab? Someone told me in the past that wrk is much more reliable and gives more consistent results.
@fat some months ago we created and used Benchy: A benchmark tool . You might find it useful to play around different benchmarks. Let me know if you play with it and if there is any feedback.
thanks for your suggestions,
yes, i agree that sometime we need some scripts to help us automate the benchmark tasks.
i’ll take a look at the docs first.
Wonder if somebody should make a go “fasthttp” equivalent for crystal…though I admit I don’t know much about it and “acing microbenchmarks” is often unuseful in real life LOL.
ok, today i’ve made another test, using wrk
but i’ve only done that in a vps box, 1 core cpu,
maybe later / tomorrow i’ll make a test on i7 box (8 core)
so, here’s the source code, and results on a vps box, 1 core cpu: