Feasibility of multi-executable architecture for a large application using Crystal

[gauravshah89]

Crystal is my most-loved language and I have been using it for small personal projects on and off. I am now starting a new project that may not remain personal in the long-run and should eventually grow on to become a large and complex desktop app. Run time performance will be a key requirement for this application.

Since crystal has higher compile times compared to other languages, what is the best way to create very large and performant applications using crystal without getting bogged down by compile times?

My Strategy => Split the application into smaller executables built from shared libraries rather than one monolithic Crystal binary. Rough proposed file structure given below:

common/ api/ worker/ cli/ will be a standalone Crystal app/lib (crystal init app/lib) that compile quickly and will be linked to each other.

large_application/

• common/
  • shard.yml # shared library (types, utils)
  • src/
• api/
  • src/
  • shard.yml # depends on ../common
• worker/
  • src/
  • shard.yml # depends on ../common
• cli/
  • src/
  • shard.yml # minimal dependencies

Is this a scalable approach for developing and maintaining serious production applications in Crystal? If this does not work out, I will have to use Rust which I don’t prefer.

Thanks for reading this long question.

[kojix2]

The short answer is NO.
Please read the conversation between Ary Borenszweig and Beta Ziliani.

Embeddable / Interoperable with ruby

Let’s take this code: def foo(x) if x.is_a?(Int32) 1 else 2 end end foo(1 || 'a') foo('a' || 1) We can compile it like this: crystal build --prelude=empty --emit llvm-ir foo.cr That will give us the LLVM IR for that file, and specifically for what the generated foo function is. How does the generated code check if x.is_a?(Int32)? The answer is that Crystal gives each type a unique ID: an integer. Here’s the LLVM IR code for the generated foo function: define internal i32 @"…

If you enjoy long, long threads, please read this one too.

Incremental compilation exploration

Happy new year! I believe that, at some point, incremental compilation will have to be introduced to Crystal in order for it to keep growing. I started exploring this problem, just a bit. I wrote a blog post about it: Incremental compilation for Crystal - Part 1 - DEV Community

[straight-shoota]

You can build multiple executables. But putting shared Crystal code in a dynamic library is not feasible. There are several hurdles to that.

It’s possible to build entirely separate executables, i.e. each one will compile the parts of the common code that it uses.

Whether that makes sense depends on the specifics of the project. It can be helpful to reduce build times. But the more Crystal code is common, the less time you’ll be saving.

An alternative to entirely separate executables would be to use compiler flags for disabling features (and thus excluding code) for development builds.

---

Do you have any concrete experience that Crystal compile times would be prohibitively unacceptable for your project? Or is this just speculation?

There are some quite large applications written in Crystal. And while compilation speed is certainly a concern, it’s usually not that dramatic. You can live with it and still be productive.

The alternative you mentioned, Rust, is not exactly known for lightning fast compilation either. So, not sure if that would do you much good.

After all, Crystal and Rust are different languages and you should choose the one that makes most sense for your project.

I can’t make any concrete promises, but we expect to improve compilation speed in Crystal with future enhancements.

[ysbaddaden]

TLDR: we can’t compile shared libraries in Crystal — unless you completely give up on the stdlib and go through a fun API + lib bindings in common.

Splitting executables is only a solution if the common code is the insignificant part, otherwise you’ll multiply the compilation times for each executable.

We still encourage you to try and compile (and recompile) large Crystal projects, and see how slow it really is in practice.

Who knows, you might spend more time compiling (that’s very possible), but you might also program faster (that’s also possible)

[gauravshah89]

I do not have empirical or experiential data on crystal’s compile times. Just that I have read that crystal is not suitable for large projects.

Using compiler flags to toggle features looks like a good option. Will go ahead with this.

Thanks everyone for the replies and suggestions.

[kojix2]

Even when you compile several Crystal programs in parallel to make the build faster, it’s better to be a bit careful.

For example, you might try to build all the crystal files in the examples directory with this command:

find examples -maxdepth 1 -type f -name "*.cr" \
  | xargs -P 8 -n 1 crystal build

This command often fails.
I’m not completely sure why, but I think it may happen because of cache file conflicts during the build.

According to DeepWiki, it might help to give each process a different CRYSTAL_CACHE_DIR environment variable.

[npn]

Just split your project to multiple binaries. Sure the shared parts would be recompiled multiple times, but overall the upsides outweigh the downsides:

• you can develop and deploy each part of the system separately
• the compilation time will be reduced greatly thanks to smaller code size

rethink about your application, and split the code accordingly:

• some services can be called using shell command, there is no need to include as source code level
• some services can be run in separated processes, they can still share the same settings if you use the right libraries

I do this in practice and it works well for me for several years already.