Very slow build speeds for hello world

We have an internal discussion open for it with several proponents, but it hasn’t been approved yet.

It took years for them to approve Rust, too. I think they only approved it last year or maybe 2022, and that’s with big vocal support from the Ruby maintainers (many employed by Shopify on a sister team to mine) for Rust extensions and YJIT.

2 Likes

I think this is only somewhat true; faster is always going to be better, and should be the aim - though everyone (even Go devs, from a quick search) will always complain it’s not fast enough.

Is there something about what I was quoting that says otherwise?

I think your reply to that quote just doesn’t acknowledge that faster compiles are always going to be better (yet, also never enough), and should be the aim - hence my post.

Do you disagree?

You said “this is only somewhat true”, which implies that at least one aspect was false. That’s what I’m addressing here.

If you want to add to the discussion, you can do it without that sort of implication.

Yes, I did say that, and my reply does add to the discussion about this. It seems like you’re mostly addressing my post’s tone instead of its actual content; so to be clear, I’m obviously not saying you’re lying or even wrong. Partial-truth in retrospect may seem like a correct but an overly aggressive choice of words from me.

So imho -

  • Yes, it’s true that it’s that dev-speed can make slower compile times more acceptable (your point, and @Xen’s, I think)
  • and Crystal should always aim to be faster to compile
  • and it’ll never be fast enough
  • and a 5 second hello world compile is wild

Without the other points, it could be easy to just accept the current state of compile speed and not improve.

If I were to say your reply was condescending, that would be about its tone. Your post contains the words “only somewhat true”. I have been, quite literally, addressing the accuracy of its content from the beginning.

I asked you to clarify because it was not obvious.

Partial truth (also commonly known as a “half truth”) means part of the statement is untrue. That’s not a reaching interpretation, that’s what that phrase mean both in common usage and in the literal definition of the words. He and I were saying that it can still be a net win over other languages — nothing more, nothing less. Nobody is saying compile times are good. Even the Crystal core team knows the compiler is slow.

That we did not make the point you wanted to hear does not make it a partial/half truth. From a purely factual perspective, your claim that it is “correct” is incorrect.

You’ve literally partially understood partial-truth, and a simple search would show you that you the one incorrect understanding - Half-truth - Wikipedia - it is explained clearly in the first paragraph.

Quote:

A half-truth is a deceptive statement that includes some element of truth. The statement might be partly true, the statement may be totally true, but only part of the whole truth

Which is exactly, and clearly how I meant it. So, if somehow it means something else in English other than from the US you have my apologies. This is how I meant and used it - correctly - with common US English. FYI, happy to take this tangent to DM.

He and I were saying that it can still be a net win over other languages — nothing more, nothing less.

I agree, yet it’s part of the whole truth, as I explained.

Would you like to address any of the points I actually made?

You do realize that that’s in the context of deception, right?

Yes. I fully understand what I wrote - both the meaning as well as the context. Also “partial-truth in retrospect may seem like a correct but an overly aggressive choice of words from me”. If you’re upset and or didn’t like my tone, I apologize.

Please move this to DM and or stop derailing this thread.

Maybe slow compile times leads to awful first/early devx. i.e. how many people had this and didn’t post a thread?

If that’s the case, the arguments you’ve made have not been coherent. Either you do think I’m being deceptive (since you say here you fully understand what you wrote) and the part you said above about you were “obviously not saying [I’m] lying or even wrong” wasn’t true, or you don’t think I’m being deceptive and “partial truth” is the wrong choice of words. But mixing the two doesn’t make any sense.

I’m not upset and it was never about your tone.

You don’t get to take shots at people publicly and tell them they should take it up with you privately.

This thread had no activity for over 2 months before you brought it back up to say I was wrong and then mince words until they had no meaning. I’m not the one derailing it. I’m merely responding to something that was incorrect.

Compilation time has been the single biggest complaint people have had about Crystal for years. There isn’t a single member of the core team who is unaware of the frustrations it causes people. There have been probably a half-dozen attempts to improve it. The interpreter was added to mitigate development latency (Ary gave a presentation about it 3 years ago), there have been discussions about incremental compilation (one of which was already linked in this thread), and there have been quite a few other approaches to solving the problem of long compilation times.

Nobody is truly happy with the state of compilation times in Crystal. We work around them, but not everyone has to say it’s a problem for folks to know it’s a problem.

The fact that you took this is as a shot at you is the whole issue here, imho. I have no idea who you are, nor anything against you. I was (apparently not) simply pointing out that whilst your statement is true, speed matters. Apparently you agree, yet don’t like how I pointed it out.

Agreed - it’s been the case for the whole time I’ve used it. In fact for me personally it’s very nearly the only complaint - even on a relatively fast computer.

Crystal is an interesting language that is statically typed yet duck typed. Slow compile time is a trade-off. I want to understand better why it is slow. I tried running the perf command, but it didn’t really help me.

Perhaps the best way would be to use print debug to show the elapsed time. However, doing this requires the Crystal compiler to compile Crystal compiler many times.

The problem is that Crystal compilation is slow, and to find out why, I need to compile Crystal with the Crystal compiler. Yes, the Crystal compiler, which is notoriously slow, is being used to compile Crystal. So, as you can imagine, the process of compiling Crystal takes a long time because, well, Crystal compilation is known to be slow.

Still, I think it’s interesting and I’d like to try it.
Is there an article that goes into detail on what is taking how long a Crystal compilation takes?

1 Like

Relex, all we love Crystal, right?

1 Like

The shot that you took at me was when you said I was being willfully deceptive with all that about partial/half-truths.

The whole issue is that you said something that was factually incorrect, I gave you an opportunity to correct it, pointed out how you were incorrect, and you dug your heels in — talking in circles, ignoring the multiple logical fallacies I’ve repeatedly pointed out to you, and writing incoherent rebuttals.

If this discussion has made you uncomfortable at any point, you could’ve simply taken a moment to reflect on what you’ve said, and acknowledged it. Or, if you didn’t want to do that kind of self-reflection, you could’ve instead said absolutely nothing. You chose to keep responding the way you did.

Yes, you said I was either incorrect or deceptive. Neither are true. Are you surprised that I wouldn’t like that?

If I were a computer genius,

  1. I would design an intermediate representation (IR) for the Crystal language.
  2. I would convert files and Shards into that IR.
  3. I would compare those IRs to quickly determine the perfect types for the entire program.
  4. I would add type information to the IR and generate multiple LLVM-IRs.
  5. I would then compile and link those LLVM-IRs in parallel, smoothly and efficiently.

However, this all assumes that types can be decided in advance as if by magic, and if that were possible, life would be much simpler, wouldn’t it?

1 Like

@russ and @jgaskins , this is a good point to continue this conversation in private, if any of you want to keep it going :pray:

6 Likes

I investigated where the Crystal compiler is spending time using classic print debugging.

module Crystal
  class Compiler
    def compile(source : Source | Array(Source), output_filename : String) : Result
      source = [source] unless source.is_a?(Array)
      # 1 new_program
      program = new_program(source)

      # 2 parse
      node = parse program, source

      # 3 semantic
      node = program.semantic node, cleanup: !no_cleanup?

      # 4 codegen
      units = codegen program, node, source, output_filename unless @no_codegen

      # 5 cleanup
      # ... omission ...
      Result.new program, node
    end
  end
end

I measured the time spent on each part of the Compiler#compile method.

  1. new_program
  2. parse
  3. semantic
  4. codegen
  5. cleanup

I also timed the external LLVM functions.

  • LibLLVM.run_passes
  • LibLLVM.target_machine_emit_to_file

See the patch file for the actual code used.

After compiling and installing the Crystal compiler with changes to display elapsed time, I compiled the unmodified Crystal compiler.

Stage Time (seconds)
new_program 0.000388207
parse 0.000065000
semantic 12.552620028
codegen 355.245409133
- LibLLVM.run_passes 252.340241198
- LibLLVM.target_machine_emit_to_file 93.280652845
cleanup 0.000013180
total 367.798495548

The results show that the two LLVM functions consume most of the time (93.97%).

This fact might be common knowledge for those who have been using Crystal for a long time or are part of the core team, but for someone like me who started using Crystal later, it is very interesting.

1 Like

wait, it seems to me what have you just tried is just the output of compile a program with -s flag though?

If you compile a program with -p,

-p, --progress                   Enable progress output

You can found 90%+ time spent on codegen, if use --release option, with more.

2 Likes