Sponsorship for Incremental Compilation

if your business logic is stable, port it to native.

but if you want to move fast, keep rails is the best.

2 Likes

FWIW: I often compile my programs twice. The ā€œplainā€ build, and the ā€œā€“releaseā€ build. The plain compile gives me an object file with all the extra debug info in it, and also compiles quickly. The ā€œā€“releaseā€ build gives me the object file which has all the optimizations figured out. For everything Iā€™ve written, that plain build is quick enough that I have no complaints with the speed.

1 Like

@pynixwang Yes, I think Iā€™m currently more productive with Rails (and all of the available Ruby gems), especially I have a lot more experience with Ruby and Rails. The idea is to contribute to the Crystal language and hopefully make things even faster and more productive in the future. Iā€™m fully aware that this is a very ambitious and long-term goal, and it might take 10-20 years. (But hopefully less than that!)

@drosehn Thatā€™s a good point about the ā€œā€“releaseā€ builds taking longer than the debug/plain builds. Iā€™m actually not too concerned about the compile times for release builds, and Iā€™m more interested in incremental compilation for the plain builds. Itā€™s very important that Crystal can scale up to very large codebases and be adopted by huge companies.

Itā€™s also just a very interesting technical challenge, and itā€™s a great way to learn more about the Crystal language and compiler. So it would be a lot of fun to work on, just for its own sake, and just to see if itā€™s even possible.

1 Like

I follow this subject. Iā€™ve an work in progress app with ~15k of lines of code plus some libraries so about 25k lines of code on top of the stdlib, here below my results on i7-8750HQ 6 cores:

> make build_server
/usr/local/bin/crystal build  src/endpoints/server.cr -o build/server
(...)
A total of 1 warnings were found.
user: 17.22 system: 5.18 elapsed: 0:12.66 CPU: 176%

17sā€¦ honestly, itā€™s a bit slow, still doable but as ndbroadbent, it is currently my main concern for using or not Crystal in futures paid developments.

In the end, the compile time is what slows me down the most (with JSON boiler plating but it has been fixed by me lol). The language itself and the compile-time error check is just pure pleasure :slight_smile:

1 Like

Since weā€™re sharing compilation times:
241.361 LoC, 2.658 files, on i7-8700
3 minutes and 50 seconds compiled normally
5 minutes and 52 seconds with --single-module (the only way it works after compilation anymore)
60 minutes with --release and --single-module

Uh-oh, thatā€™s a lot. Thanks for sharing!

By any chance you have big hashes in the source? There is a know issue with that, can be part of the reason. Also there is a workaround.

It would be great to have a simple way to profile compilation.

We have a Rails app about this size, having to wait 3-5 minutes instead of 3-5 seconds just to refresh the page wouldnā€™t work.

Big hash literals, yes, quite a few. Also many tuples of various sizes.

@Exilor You can try this workaround, which can help with compilation speed for you then https://github.com/crystal-lang/crystal/issues/5792#issuecomment-371672149

Interesting!

@asterite regarding your comment Unicode tables as constants Ā· Issue #4516 Ā· crystal-lang/crystal Ā· GitHub

The codegen phase needs to be smarter about this (avoid allocas, generate less code), but for now this workarounds works well.

What would be the way to do this then? (How to avoid llvmā€™s alloca?)

I tried that but nothing changed except the binary becoming slightly smaller.

Thatā€™s confusing, why would the binary size change? Youā€™re speaking about defining method like https://github.com/crystal-lang/crystal/issues/5792#issuecomment-371672149 and using it instead of literals, right?

I had many classes with hash literals of various sizes assigned to constants. What I did is have the constants be empty and then fill them up later at runtime, which should be the same thing.

This would be super easy, but wouldnā€™t that make the compilation take just as long as it does now anytime anything changed?

This is a really good point, the parts that take a long time come towards the later phases