# Incremental compilation exploration **URL:** \ **Category:** Crystal Contrib\ **Created:** [January 3, 2023, 2:28pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224 "2023-01-03T14:28:37Z")\ **Posts on this page:** 20\ **Page:** 5
**Author:** ![Blacksmoke16](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/blacksmoke16/32/1241_2.png) [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)\ **Post date:** [August 11, 2023, 1:09pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/86 "2023-08-11T13:09:26Z")
> [@zw963](#): > > in this form, when newcomer ask questions, probably need several days to get the answer, because no one dedicated for this, i feel like the core members really don’t have time for this. I’m pretty sure this isn’t true. There aren’t many, if any, help threads that go unanswered for more than a day, if that. Answering questions and helping users out is my main task. ---
**Author:** ![jwoertink](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jwoertink/32/50_2.png) [@jwoertink](https://forum.crystal-lang.org/u/jwoertink)\ **Post date:** [August 11, 2023, 3:13pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/87 "2023-08-11T15:13:31Z")
meanwhile, my app at 33k loc takes ~ 11 minutes to compile in release 😂 ---
**Author:** ![Ragmaanir](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/ragmaanir/32/1174_2.png) [@Ragmaanir](https://forum.crystal-lang.org/u/Ragmaanir)\ **Post date:** [August 11, 2023, 3:38pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/88 "2023-08-11T15:38:07Z")
Could you give some more details? Whats you CPU? Does the 33k loc include the used shards? What does `--stats --progress` show. How long does it take without `--release`. Do you use a lot of macros? Is the app on github? ---
**Author:** ![jwoertink](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jwoertink/32/50_2.png) [@jwoertink](https://forum.crystal-lang.org/u/jwoertink)\ **Post date:** [August 11, 2023, 3:56pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/89 "2023-08-11T15:56:10Z")
The app isn’t open source. Without release is like 2 minutes. The 33k loc is what I’ve written, so without shards, and there’s a TON of macros lol 😆 Pretty much how Lucky is designed I think focuses in all of the pain areas for compilation lol. Like, Lucky uses the `run` macro a lot, tons of Unions and blocks, many methods without type signatures due to gnarly Hashes or whatever. ---
**Author:** ![Devonte](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/devonte/32/1927_2.png) [@Devonte](https://forum.crystal-lang.org/u/Devonte)\ **Post date:** [August 11, 2023, 5:44pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/90 "2023-08-11T17:44:12Z")
> [@zw963](#): > > Instead of talking about not having time for these features, need do more research, how about do something making Crystal acceptable to larger companies? bring the founders back? make community members more willing to get involved? I feel like this detracts from the original point of this thread. Incremental compilation isn’t going to resolve any of those points, and this thread _is_ the research “stage” of things as it were, which is exactly what is needed for progression. ---
**Author:** ![botato](https://avatars.discourse-cdn.com/v4/letter/b/a88e57/32.png) [@botato](https://forum.crystal-lang.org/u/botato)\ **Post date:** [August 12, 2023, 6:10pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/91 "2023-08-12T18:10:00Z")
Personally I agree with @MistressRemilia that my ideal for development would be improving the interpreter and editor integration towards something like emacs slime or sly mode for lisp. We don’t really care about times of release builds, just as long as builds are fast during development. If we can develop fully with the interpreter, we don’t care about times of debug builds either, as we can make changes to our program without even having to restart it and lose existing state, let alone wait for a recompile. Plus an interpreter has many possibilities for runtime introspection that just can’t be done or would be much harder with the compiler. For example, an object inspector to browse the runtime values of variables, being able to trace functions, inspect their inputs and outputs, play with those objects from the repl, view backtraces and more of all running fibers, redefine existing classes and methods to make changes without restarting, etc. Maybe full type inference is still too slow for the interpreter, in that case I’d be fine with just having the interpreter give runtime type errors, as with this style of interactive development runtime is effectively compiletime. And still could have lsp running in background that would be running the compiler and alerting on type errors the compiler detects some seconds later. Of course incremental compilation would be awesome also, but it sounds like it may be very difficult or even impossible without sacrificing some of what makes crystal lang special, and a solid interpreter and editor integration with it would mostly solve the same issues around fast development feedback cycle, in a potentially even better way. ---
**Author:** ![kk723](https://avatars.discourse-cdn.com/v4/letter/k/ac8455/32.png) [@kk723](https://forum.crystal-lang.org/u/kk723)\ **Post date:** [August 13, 2023, 8:29pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/92 "2023-08-13T20:29:06Z")
> [@asterite](#): > > The questions we need to ask and answer (the core team, but mainly the community) is what’s the appeal of Crystal. Is it mainly the standard library? Is it being able to prototype without specifying types in methods? Is it the concurrency model? Is it macros and compile-time reflection? What if we require you to require all file dependencies up-front? That’s a really interesting question. I’m interested in Crystal because I see it as a super-fast version of Ruby. I want to keep the same way of writing code as Ruby, but have it work much faster. Basically, I want programs to run really quickly. Could I use TruffleRuby? Yes, but then I have to distribute the GraalVM along with my code. This becomes problematic on IOT devices or applications that need a really smaller footprint. My use case is for IOT devices. Sort of like Chef but for IOT. Distributing Cruby with RBENV or distributing GraalVM for TruffleRuby just adds another layer of maintenance and version management. This is where the idea from Go sounds nice, where you can “write once, run everywhere”. Here’s a post from that explains that idea more. > Go compiles to a target OS and CPU architecture. The result is statically linked so the binary will run almost anywhere that those two values hold true. This is better than the old C/C++ days where you had to make sure the right libraries were installed for dynamic loading (although static linking has always been available, it was costly in the executable size, which isn’t as bad as it used to be). But the most important thing for me, the absolute most important thing, is Ruby syntax with the execution speed of a statically typed language. > [@botato](#): > > We don’t really care about times of release builds, just as long as builds are fast during development. I agree. Slow compilations times directly affects programmer productivity, which is a business expense. Companies still have to pay for compile times. So yes, vastly reduced compile time during development is critical IMO. If you’re thinking about incremental compilation. Be sure to give us the option to clear the cache. I’ve seen this many times where we just simply need to clear the cache. ---
**Author:** ![mavu](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/mavu/32/213_2.png) [@mavu](https://forum.crystal-lang.org/u/mavu)\ **Post date:** [August 14, 2023, 2:32pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/93 "2023-08-14T14:32:35Z")
> [@tsornson](#): > > The two crystal changes that I could see helping this (_both_ suggestions were provided above, and I think both would be needed) are: > > 1. A new crystal tool that adds missing typing information to methods after type inference is complete (would operate similarly to the existing `format` crystal tool in my mind) to quickly add typing everywhere it’s needed. > 2. A new compiler flag that would fail fast whenever it encountered a new signature that was missing typing information (rather than having multiple “tiers” of compilation speed, I’d prefer just to have a one-and-done approach). > > I think with these, crystal would be a viable approach for larger projects going forward. It wouldn’t help with the LSP problem for smaller projects as much. I wonder if there’s a different approach that could be used for that? Hello Megathread©, I would like to draw some attention to this comment quoted above, because I feel (or overlooked) it was pretty much ignored, but as far as I can tell offers basically the perfect solution to this problem. Keep the language as is, but offer a tool that can automatically change the source to include all infered type information. The compiler can then switch to a incremental compilation mode when it detects that all types have been pre-infered in the source. The user would only need a way to manually re-do the type infering thing and the compiler would need an error message to suggest when this is neccessary. –release always does a full re-annotation. Other than that, I would add that my personal opinion and use of crystal is not affected by compile times at all. Taking the occational minute to have a drink, or look out the window, is a very welcome thing for me. But good editor support is mandatory for this, so you can at least be reasonably sure that you produce syntacticaly correct code. ---
**Author:** ![kojix2](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/kojix2/32/1583_2.png) [@kojix2](https://forum.crystal-lang.org/u/kojix2)\ **Post date:** [August 14, 2023, 3:45pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/94 "2023-08-14T15:45:20Z")
This is an interesting idea, but I don’t think it is as perfect as it first seems for the following reasons. How are you going to “manually re-do the type inference”? All source code has typing information. So what we have now is just illegal code with an illegal argument type. The compiler cannot distinguish whether it is a human error or a redo case. But we have Git. what if we use Git to revert the code back to its not-yet-typed state? As you know, Git stores past snapshots in a .git directory. This mean the approach of storing the “untyped” canonical code separately. So it is essentially the same approach as Ruby does. Save code and type files in different locations, I think. ---
**Author:** ![mavu](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/mavu/32/213_2.png) [@mavu](https://forum.crystal-lang.org/u/mavu)\ **Post date:** [August 14, 2023, 5:11pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/95 "2023-08-14T17:11:08Z")
not sure if i like the idea of involving git. not everyone uses git, it introduces an additional dependency, and most importantly, it just does not feel right 😃 The idea is not to have additional/alternative files. The idea is that there are 2 “modes” you can use crytal in. One is the default, as it is now. The other is “every type must be specified” but there is a tool that helps you initially to go from default → explicit, and you gain the possibility of incremetal compilation. while you continue to work on the code base, you either provide the type information yourself (after you decided to go this route by running the “annotator”) or just write plain crystal and re-run the tool to have the code re-typed. ---
**Author:** ![Xen](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/xen/32/2288_2.png) [@Xen](https://forum.crystal-lang.org/u/Xen)\ **Post date:** [August 14, 2023, 6:28pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/96 "2023-08-14T18:28:34Z")
> [@mavu](#): > > The other is “every type must be specified” but there is a tool that helps you initially to go from default → explicit, and you gain the possibility of incremetal compilation. Problem is, it’s bit of a one way street. Once the tool adds in the missing types, you can’t go back. Using a second file for automatic types would solve that, and the next idea would be to run it automatically and cache the result rather than writing it to the original source file. But as I understand asterite, the problem is that as soon as you fiddle with a type somewhere, it can cascade through the whole application, so you basically have to clear the whole cache. So whether the inferred types cached or burned into the source, you’ll end up in a situation where touching types will make Crystal throw type errors for half your edits, requiring you to clear the cache/rerun inferrence anyway. Regarding the ergonomics of no types, a practical example: Kemal has a handy redirect method: ```auto def redirect(url : String, status_code : Int32 = 302, *, body : String? = nil, close : Bool = true) @response.headers.add "Location", url @response.status_code = status_code @response.print(body) if body @response.close if close end ``` One compile error later and I was wondering why on earth it doesn’t take an `URI` (maybe it predates the `URI` class?). One could argue “why not a `Path`?” too, if so inclined. Of course this can be fixed with a union, but the point is that the author must know ahead of time what types could be useful. Or, they could go the other way and say “We’ll redirect to anything that replies to `to_s`”. Of course, if they’re really hooked on the type thing, making it just take `URI` would be the obvious answer. Which would be mightily annoying if you happen to have a `String`. ---
**Author:** ![mavu](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/mavu/32/213_2.png) [@mavu](https://forum.crystal-lang.org/u/mavu)\ **Post date:** [August 14, 2023, 7:23pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/97 "2023-08-14T19:23:56Z")
> [@Xen](#): > > So whether the inferred types cached or burned into the source, you’ll end up in a situation where touching types will make Crystal throw type errors for half your edits, requiring you to clear the cache/rerun inferrence anyway. Thats the case for every strictly typed language, right? Change one type somewhere, change everywhere it’s interacted with. It would still eliminate a lot of compile time, because changes where you constantly change the type of things is probably only a small percentage of the changes you do. The main problem with this approach is how to handle the standard library. I think it’s source files would need to be rewritten for this approach too.(not exactly sure here) That is not such a big problem in itself I think, but is certainly a bit of a weird situation. You would have a directory in your source tree that contains the modified standard library source files of the language itself. Weird thought. but why not? ---
**Author:** ![npn](https://avatars.discourse-cdn.com/v4/letter/n/e68b1a/32.png) [@npn](https://forum.crystal-lang.org/u/npn)\ **Post date:** [August 14, 2023, 7:48pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/98 "2023-08-14T19:48:16Z")
Full type restriction only works for tools like LSP, so we can have nicer auto completion. But it won’t help for incremental compilation, as most type restrictions are just abstract types: parent types, module types, virtual types, generic etc. Generate the method definitions for all the possible types will be a huge bloat. But I think we can sidestep the problem by introducing some tangible interfaces/header modules. And we do not need this feature for all the code, just when you need it. For example you have a markdown compiler, that take a markdown string and output a html string. We do not care about what is going on inside the Markdown class, we only care about the output html. Reanalyzing/recompiling the content of Markdown module is mostly a waste of time. Even if the code is changed, as we only care about a single entry point, the analyzing/compiling of Markdown module can be doing separately from the main pipeline. For the sake of making the comment easier to follow, from now I will refer to the interface/header/module simply as unit. There is some problem with this approach though, because Crystal has a huge runtime that only get enabled when some code required it (for example the PRCE2 engine), separating the requirement needed for each “unit” will be hard. But the more I think about it the more possible it becomes: So what if we wrap all the runtime dependencies into some unit itself. for examle unit PRCE will have a public method called `.init_runtime` that will take care of all the logic related to the initialization, so even multi units depends on PCRE and call .init\_runtime multiple times the actual initialization will only get called once. This also apply to top level constants as we can just wrap them inside some units. NOTE: unlike the traditional C/C++ headers, we do not need hundred of seperated units, just a handful of them. The idea is somewhat similar to [RFC: Declarative Compilation Units · Issue #13637 · crystal-lang/crystal · GitHub](https://github.com/crystal-lang/crystal/issues/13637), but instead of using annotation, we explicitly declare some units in the place we need them. For example the markdown above: ```crystal unit MD require "markdown" fun to_html(input : String) : String Markdown.to_html(input) end end puts MD.to_html("example") ``` as `MD.to_html` takes a `String` and returns a `String`, the whole unit can be processed separately and cache independently. There might exist type mismatch, but I think some mechanism similar to `.from_unsafe` and `#to_unsafe` can be introduced to convert the types between units. Though this issue needs to be invested further. ---
**Author:** ![Xen](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/xen/32/2288_2.png) [@Xen](https://forum.crystal-lang.org/u/Xen)\ **Post date:** [August 14, 2023, 9:55pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/99 "2023-08-14T21:55:05Z")
> [@mavu](#): > > Thats the case for every strictly typed language, right? > Change one type somewhere, change everywhere it’s interacted with. Yeah, and you’ll be following the cascade around fixing up types. With duck-typing you’ll primarily be dealing with where it doesn’t quack right. ---
**Author:** ![Devonte](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/devonte/32/1927_2.png) [@Devonte](https://forum.crystal-lang.org/u/Devonte)\ **Post date:** [August 15, 2023, 1:00am UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/100 "2023-08-15T01:00:57Z")
> [@Xen](#): > > So whether the inferred types cached or burned into the source, you’ll end up in a situation where touching types will make Crystal throw type errors for half your edits, requiring you to clear the cache/rerun inferrence anyway. This can be easily solved by caching a dependency graph of the states in the program and only updating/caching the the parts that are affected. ---
**Author:** ![Devonte](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/devonte/32/1927_2.png) [@Devonte](https://forum.crystal-lang.org/u/Devonte)\ **Post date:** [August 15, 2023, 1:19am UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/101 "2023-08-15T01:19:11Z")
This approach, even as a proof of concept, doesn’t feel right to me. Leaving declarative compilation units to the end-user doesn’t make sense for something that the compiler should handle. It’s also very easy to run into conflicts with projects that reopen types. For example, lets say `String` is declared as a unit in the standard library, but a shard reopens it as a class, is it still processed as a unit? Would that count as a compiler error like reopening structs as a class or vise-versa? The annotation version has a similar issue: if `String` isn’t declared as a compilation unit in the standard library, but _is_ in a shard, does the shard get priority and thus processed separately? This becomes increasingly complex when you factoring multiple shards with conflicting monkey-patched code. ---
**Author:** ![npn](https://avatars.discourse-cdn.com/v4/letter/n/e68b1a/32.png) [@npn](https://forum.crystal-lang.org/u/npn)\ **Post date:** [August 15, 2023, 2:26am UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/102 "2023-08-15T02:26:05Z")
Classes defined in a unit have different namespaces. They will not overide the standard one. Heck, to be extra sure we will wrap the standard types that is required inside unit as separated types, too. Every exchange between units and units vs normal code will involved some implicit casting. Also, class reopen/semantic changing is still something that need to be addressed for every incremental compilation solution out there, I don’t know why you think it is a unique problem for this proposal. How about for the time being just refuse to compile if the types are not compatible? Remember this is some opt in feature, not opt out. And we still need to decouple/modularize the runtime dependency it we want to expand crystal backend targets anyway. Like changing the GC, replacing the regex engine, custom concurrency mode etc. ---
**Author:** ![Devonte](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/devonte/32/1927_2.png) [@Devonte](https://forum.crystal-lang.org/u/Devonte)\ **Post date:** [August 15, 2023, 3:18am UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/103 "2023-08-15T03:18:03Z")
I still don’t understand why this should be a thing that end-users must do explicitly, not to mention that this wouldn’t be compatible with most of Crystal’s ecosystem unless they are updated to be (which is unlikely as it is). > [@npn](#): > > Also, class reopen/semantic changing is still something that need to be addressed for every incremental compilation solution out there, I don’t know why you think it is a unique problem for this proposal. I don’t, but it’s much more manageable with the other solutions compared to this one which has the present conflicts. I don’t see what benefits this solution brings compared to the others. In fact, there are presently more downsides pointed out by @yxhuvud in the GitHub issue. ---
**Author:** ![npn](https://avatars.discourse-cdn.com/v4/letter/n/e68b1a/32.png) [@npn](https://forum.crystal-lang.org/u/npn)\ **Post date:** [August 15, 2023, 7:26am UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/104 "2023-08-15T07:26:49Z")
To be honest even I do not fully believe that it works. the problem is I don’t see any other viable approach, at least for the current Crystal language. In Crystal, fully typed is just an illusion, for example this piece of code ```crystal module Foo def bar(baz: String) : String # do something that returns a String end end ``` This seems to have enough information to compile separately, yeah? But in fact we don’t know what `String` is until it gets called from some entry point. Even at the top level, the definition of `String` can still be changed somewhere else, detecting the changes and invaliding the cache will still be a very hard problem. And even if there are some genius that can solve the incremental compilation problem with the current Crystal design, there are still many remaining problems. For example to keep the magic works, the whole AST tree need to be kept in the memory, and need to be reanalyze all the time. This is a huge problem when you project grows bigger. So, we still need to introduce some rigid interfaces with strict restrictions so it can be detached from the main pipeline. How strict is it, I think restricting to the types that implement `.from_unsafe`, `#to_unsafe` and those that can be converted from literals for starter. There will be concern about conversion overhead, but even nowadays we still do it all the time calling methods from C libraries. A smart enough compiler can skip the casting if it determines the types are compatible, and if you are confident enough, you can just do `unsafe_cast` like usual. What I like about using `unit` is that for critical code you can just reopen the unit, implement the logic (no overhead here because they are in the same domain), then add an entry point to be called from outside. > I still don’t understand why this should be a thing that end-users must do explicitly the stdlib can implement it own units, as the libraries, but I like that user can write one explicitly, because I think we only need a handful of them, because of the obvious reason: the potential conflicts between types. it is better to check/resolve the conflicts when there are only a dozen of them, instead of thousand. ---
**Author:** ![zw963](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/zw963/32/1623_2.png) [@zw963](https://forum.crystal-lang.org/u/zw963)\ **Post date:** [August 16, 2023, 3:54pm UTC](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224/105 "2023-08-16T15:54:24Z")
I have exactly the same thoughts with you! [Previous page](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224.md?page=4) [Next page](https://forum.crystal-lang.org/t/incremental-compilation-exploration/5224.md?page=6)