Slow compilation time inside a Docker container

Help & Support
1

Anyone have any ideas why my compilation times are an order-of-magnitude slower when running inside a Docker container?

Using the same flags – eg. shards build <prog> --production --release the code compiles in about 3 seconds on my local MacBook Pro (with CRYSTAL_WORKERS=20) and about 3 minutes on Google Cloud Build (using a 32-CPU machine with CRYSTAL_WORKERS=32).

I’m using the official base image (FROM crystallang/crystal:1.7.3) and same version of crystal locally on my mac.

Any thoughts?

2

Well for one Crystal’s compilation process is single threaded at the moment, so pretty sure CRYSTAL_WORKERS=20 part isn’t actually doing anything. Also because it is single threaded, single core perf is more important than how many cores. So it’s possible/likely your laptop just has faster single core perf than GCB. Cache may also play a factor since GCB might be going from scratch each time versus when on your laptop it can reuse some parts.

Try building with like --stats --progress and see what part is taking longer.

3

In release mode, yes. In dev mode it will build every file separately in parallel and then link them. Type resolution is still single threaded though, even in that case.

And it is very likely it is the cache that is the main factor. Try clearing it locally to see the difference.

1 Like
4

Ahhh fascinating. Very helpful thank you George. I cleared the cache locally and it’s now more like 30 seconds locally vs 90 seconds on the server (I was off in my 3 minute estimate – turns out actual data is helpful!). Still a significant difference but a few less OOMs ;)

Here’s the output

local (M1 MacBook Pro):

Parse:                             00:00:00.000075583 (   0.77MB)
Semantic (top level):              00:00:00.315950583 ( 125.28MB)
Semantic (new):                    00:00:00.001477542 ( 125.28MB)
Semantic (type declarations):      00:00:00.020243833 ( 125.28MB)
Semantic (abstract def check):     00:00:00.008588708 ( 141.28MB)
Semantic (restrictions augmenter): 00:00:00.006174125 ( 141.28MB)
Semantic (ivars initializers):     00:00:00.011086917 ( 157.28MB)
Semantic (cvars initializers):     00:00:00.116850291 ( 157.33MB)
Semantic (main):                   00:00:01.100900083 ( 414.20MB)
Semantic (cleanup):                00:00:00.000392166 ( 414.20MB)
Semantic (recursive struct check): 00:00:00.000914292 ( 414.20MB)
Codegen (crystal):                 00:00:00.652138583 ( 478.20MB)
Codegen (bc+obj):                  00:00:30.467765666 ( 478.20MB)
Codegen (linking):                 00:00:00.116556292 ( 478.20MB)
dsymutil:                          00:00:00.122026875 ( 478.20MB)

Codegen (bc+obj):
 - no previous .o files were reused

Docker container (GCP cloud build - 32 CPU builder):

Parse:                             00:00:00.000200370 (   1.05MB)         
Semantic (top level):              00:00:00.942076548 ( 124.73MB)              
Semantic (new):                    00:00:00.003740611 ( 124.73MB)   
Semantic (type declarations):      00:00:00.046613005 ( 124.73MB)
Semantic (abstract def check):     00:00:00.027329527 ( 140.73MB)
Semantic (restrictions augmenter): 00:00:00.013416391 ( 140.73MB) 
Semantic (ivars initializers):     00:00:00.387768479 ( 140.79MB) 
Semantic (cvars initializers):     00:00:00.086807594 ( 156.79MB)           
Semantic (main):                   00:00:03.963297330 ( 397.66MB)           
Semantic (cleanup):                00:00:00.000766868 ( 397.66MB)
Semantic (recursive struct check): 00:00:00.002500417 ( 397.66MB)        
Codegen (crystal):                 00:00:01.865049806 ( 493.66MB)          
Codegen (bc+obj):                  00:01:19.350028859 ( 493.66MB)              
Codegen (linking):                 00:00:00.316829288 ( 493.66MB)
Codegen (bc+obj):
 - no previous .o files were reused

Just for kicks I ran it locally in Docker as well…

Parse:                             00:00:00.009627042 (   1.05MB)
Semantic (top level):              00:00:03.406946793 ( 124.72MB)
Semantic (new):                    00:00:00.011170333 ( 124.72MB)
Semantic (type declarations):      00:00:00.189165542 ( 124.72MB)
Semantic (abstract def check):     00:00:00.089279792 ( 140.72MB)
Semantic (restrictions augmenter): 00:00:00.026839750 ( 140.72MB)
Semantic (ivars initializers):     00:00:00.718217251 ( 140.78MB)
Semantic (cvars initializers):     00:00:00.335297166 ( 156.78MB)
Semantic (main):                   00:00:08.406600379 ( 397.65MB)
Semantic (cleanup):                00:00:00.002165583 ( 397.65MB)
Semantic (recursive struct check): 00:00:00.008630709 ( 413.65MB)
Codegen (crystal):                 00:00:06.182956086 ( 493.65MB)
Codegen (bc+obj):                  00:05:07.935315847 ( 493.65MB)
Codegen (linking):                 00:00:01.362744792 ( 493.65MB)
Codegen (bc+obj):
  - no previous .o files were reused

Net net: ~30s local build, ~90s GCP docker build, ~300s (!) local Docker build.

Here’s my theory: You’re correct in that the MacBook’s M1 is just much faster than the GCP cloud build machine, and the slow local Docker compile times are coming from the fact that Docker is running an emulation layer between ARM (M1) and the X86 Ubuntu Docker images. So in the grand scheme it’s:

Local build < Cloud build << Emulated image in local Docker

Does that make sense? Anything in the output above look off to you?

1 Like
5

There are many factors.

At first, I remember some post from asterite reveal: build on m1 almost is always much much faster than AMD64 linux laptop.

Next, accroding my own test, for same application, build on linux laptop (Arch linux) is faster(can be feel clearly) then build on local docker container(Ubuntu), I suspect it might be cause of the cache now.

6

Hi, is there any document for where the cache stored? I know ~/.cache used when build on local linux laptop, is there any other place when run in official docker？

7

https://crystal-lang.org/reference/1.8/man/crystal/index.html#environment-variables

CRYSTAL_CACHE_DIR: Defines path where Crystal caches partial compilation results for faster subsequent builds. This path is also used to temporarily store executables when Crystal programs are run with crystal run rather than crystal build. Default value is the first directory that either exists or can be created of ${XDG_CACHE_HOME}/crystal (if XDG_CACHE_HOME is defined), ${HOME}/.cache/crystal, ${HOME}/.crystal, ./.crystal. If CRYSTAL_CACHE_DIR is set but points to a path that is not writeable, the default values are used instead.

Probably should update that to also note the location on Windows.

1 Like