Add an option to disable specific modules to the crystal build command?

Community
1

In Godot, when we compile, we can optimize a build for size. This allows us to disable certain features. For example, a 2d game developer can disable all the 3d functionality. This allows for quicker compile times and a lower filesize. With Crystal, I’m not worried about filesize since it’s already pretty low and not an issue for me. However, I saw a compile time issue on reddit and it made me think of an idea! :D

I was browsing the compiler GitBook, and noticed there is nothing that lets us disable certain modules. There is quite a bit of modules/classes that my gameserver will never use ( ECR, HTML, Levenshtein, OAuth, OAuth2, Readline, URI, XML, Zip, Readline, YAML, MIME, BigRational, Atomic, Adler32, Benchmark, GC, StringScanner, WeakRef, UDPSocket, Termios, OptionParser, ZLib, Complex).

Some of these will not be loaded because I will not require them. Which is cool. However, if the developer could disable certain features that negatively affect compile speeds… that would be awesome!

Of course each developer has different needs dependent on their project, I’m not trying to say these are unnecessary (this is relative to my personal project).

Now, if some of these are required for the language to function (then I basically just shot myself in the foot with this topic). But I think it’s a worthy discussion to have regardless.

What do you all think?

2

I’m pretty sure code you’re application doesn’t use doesn’t end up in the binary. I.e. if you don’t use XML, none of the XML module code ends up in the final binary.

If filesize is a requirement you can also use --no-debug flag which would remove debug info, making the binary a bit smaller. Also could look into https://linux.die.net/man/1/strip which could make it a bit smaller again.

1 Like
3

@Blacksmoke16 No no, filesize is not an issue. I am thinking of ways a developer could give options to crystal build that removes specific functionality that increases compile times. This could be a good solution until incremental compilation maybe? The reddit thread I’m viewing (forgot to post in the OP) is here btw.

The real question is: How much speed does each class/module add when compiling? Sort those by DESC order, and then a developer can now nit-pick which ones they don’t need, and they will have faster compiler times. (This is all hypothetical, I’m just brainstorming out loud). Thank you for your patience.

4

Crystal already avoids generating code (LLVM IR) for stuff that’s not used. The parse time (inspected with crystal build --stats) is already pretty fast for the amount of code it needs to parse each time.

As Ary outlined the biggest issue is that we can’t cache a lot as everything can modify everything.

Then the second thing where we are in disadvantage compared to say Rust, is that we actually ship a fairly big runtime environment (the stuff doing async IO and fiber management) even for the most simple puts "Hello World". Third the union type system easily leads to an exponential explosion in the amount of types we need to handle and generate code for, especially in conjunction with also supporting generics. We can’t generate an Array class but have to generate one that handles Array(String) another one for Array(Int32) and another one for Array(String|Int32) and so forth. In summary this generates a lot of LLVM-IR that LLVM then has to churn through for each build.

2 Likes
5

@jhass Thanks for your insight!!

Is it possible to compile third party shards, so they don’t have to be compiled on each compile? For example, I’m using:

require "db"
require "mysql"

Which I’m assuming each time I run crystal project.cr, the syntax in those shards are read and parsed. Is it possible to compile them separately and somehow insert that into Crystal’s core dynamically (like how require "socket") works?

6

No, that’s the point Ary is trying to make when he’s saying we can’t cache things. I can do

require "db"
class DB::SometingShippedInTheDBShard
  @my_var = "hello"
end

and because of that the compiler has to regenerate the type every time. Same for socket , stdlib is no different here, it’s just a “shard” shipped with the compiler and every program implicitly requires https://github.com/crystal-lang/crystal/blob/master/src/prelude.cr (unless overwritten with the --prelude option).

There essentially is no “baked into Crystal core”, just a few special semantic rules for some types like Tuple.

1 Like
7

Awww light-bulb moment when I read that part. I understand now. Hmmm

Edit: Just to add, I don’t want this topic to be viewed as a negative of the compiler. I was just curious if maybe there was a way to decrease the compile times by disabling specific functionality (to help that redditor).

8

–no-debug will also make compilation faster.

1 Like
9

I was reading the reddit thread, and a user posted:

Would it be feasible to implement optional concrete type annotations to speed up the compiler ( compiling ) ?

What does this mean? I know what type annotations are, but what are concrete type annotations, and how would it speed up the compiler?

That sounds like a very interesting phrase to me. Does this mean Unions are making the language not 100% statically typed, but partially dynamic, which ultimately hinders the compile speeds and nullifies the opportunity for incremental compilation?

What if… there could be a 2nd version of Crystal that doesn’t have Unions, is 100% statically typed, has incremental / modular compilation, ability to precompile String (as asterite stated), etc. Which would be available for more advanced developers who think they can live without Unions and whatnot? This would be very interesting, I’d test the crap out of it ahaha. Probably easier said than done though…

10

Does this mean Unions are making the language not 100% statically typed, but partially dynamic, which ultimately hinders the compile speeds and nullifies the opportunity for incremental compilation?

No, you will never find a union at runtime. It’s strictly a compile time thing. A particular variable can be a String or an Int32 during compilation but it will only be of one type while the program executes.

What if… there could be a 2nd version of Crystal…

Developing the language is already a massive endeavor as it is. Also without unions we might as well use Swift instead.

11

Hmm. I don’t think that’s true in practice. Because of is_a?

If an if 's condition is an is_a? test, the type of a variable is guaranteed to be restricted by that type in the then branch.

if a.is_a?(String)
  # here a is a String
end

if b.is_a?(Number)
  # here b is a Number
end

Is run when the program executes. a and b could be union types. If that variable just had 1 type instead of being a union type, it would only need 1 if condition.

edit: To be clear, I’m not against unions, I’m just trying to find out the culprit of the issues asterite stated on reddit (regarding compile speed), and see what can be done to decrease compile times for certain users. If unions are a big problem (“exponential explosion in the amount of types we need to handle”), then IMO it should be addressed

Another idea: Maybe a compile flag that disables unions. Or an option that tells Crystal to go full 100% static mode. If this could innately support incremental compilation, compile String, and basically all the issues addressed in this, this, and this post…

This could satisfy the developers who have slow compile times, while the other developers who don’t see it as an issue, are unaffected. IMHO, this could be a win-win.

Edit2: Actually, I changed my mind. If unions are stopping features that ultimately speed up the compiling process, I’m definitely against them.

12

:grimacing: It might be best to understand them better before you start denouncing them. Without them our lives would be quite a bit harder. I.e. you couldn’t use JSON.parse since that parses the JSON data into a union of all possible JSON types.

13

My life would prob be better, cause it would force me to use from_json :stuck_out_tongue: