CPU Compile Speed Benchmarks

jgaskins · March 8, 2024, 3:45pm

IIRC there was a discussion a few years ago where someone pointed out that compilation on Linux was faster than on macOS. Do we know if that’s still the case?

pfischer · March 8, 2024, 8:13pm

Let’s have a CPU shootout!

If anyone here has M1/2/3 macbooks and all those X3D Ryzens and modern Intels etc - let’s pick a github project to test compile times and throw up the numbers.
Do you know of a project that uses a lot of shards and is bigger (a lot of classes) and opensource? Some web framework? Lucky? Crystal itself?

Thanks! :)

notramo · March 12, 2024, 5:42pm

Web framework starter templates compile quite fast, so I would suggest a bigger project. The Crystal compiler seems like a better benchmark subject.

zw963 · March 13, 2024, 6:24pm

build on latest master 2b0b50602

delete the cache folder use rm -rf ~/.cache/crystal
unset CRYSTAL_OPTS ensure no compile-time flag was set.
cd crystal compiler folder
run git clean -fdx delete all intermediate files.
run time make release=1

result:

 ╰─ $ time make release=1
Using /usr/sbin/llvm-config [version=17.0.6]
g++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/include -std=c++17   -fno-exceptions -funwind-tables -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="2b0b50602" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1710327137"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -D preview_overload_order --release -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2

real    4m31.760s
user    4m27.682s
sys     0m4.183s

                   -`
                  .o+`                 zw963@mingfan
                 `ooo/                 OS: Arch Linux
                `+oooo:                Kernel: x86_64 Linux 6.7.9-x64v2-xanmod1-1
               `+oooooo:               Uptime: 7h 48m
               -+oooooo+:              Packages: 1134
             `/:-:++oooo+:             Shell: bash 5.2.26
            `/++++/+++++++:            Resolution: No X Server
           `/++++++++++++++:           DE: GNOME 45.3
          `/+++ooooooooooooo/`         WM: Mutter
         ./ooosssso++osssssso+`        WM Theme:
        .oossssso-````/ossssss+`       GTK Theme: Adwaita [GTK2/3]
       -osssssso.      :ssssssso.      Icon Theme: Adwaita
      :osssssss/        osssso+++.     Font: Cantarell 8
     /ossssssss/        +ssssooo/-     Disk: 885G / 2.0T (48%)
   `/ossssso+/:-        -:/+osssso+-   CPU: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics @ 16x 5.137GHz
  `+sso+:-`                 `.-/+oso:  GPU: AMD/ATI
 `++:.                           `-/+/ RAM: 7525MiB / 63094MiB

pfischer · March 14, 2024, 5:17pm

Thanks to all for the bench results! I will manage this overview results table…

Desktop Ryzens 7600/7700/7900 and 8500/8600/8700G and 9600/9700/9900X WANTED - do you have one?

CPU	OS	LLVM	Commit	User	Time	CPU Mark	Max TDP
Apple M4 Max	OSX Sonoma	19	675c68c	jgaskins	1m49s	? / ?	30W?
Apple M3 Max	OSX Sonoma	18	2b0b506	adamcarr	2m22s	36301/4757	30W?
Apple M3 Pro	OSX Sonoma	18	2b0b506	jgaskins	2m25s	24270/4714	30W?
Apple M1 Max	OSX Sonoma	19	675c68c	jgaskins	2m46s	22009/3840	30W?
Apple M1 Max	OSX Sonoma	17	2b0b506	jgaskins	3m10s	22009/3840	30W?
Apple M1 Pro	OSX Sonoma	17	2b0b506	self	3m16s	21793/3812	30W?
Intel i9-14900K	Linux 6.7.9	17	2b0b506	Blacksmoke16	3m20s	61057/4773	250W
Apple M1 Pro (*1)	OSX Sonoma	19	675c68c	daliborfilus	3m3s	21793/3812	30W?
Apple M1 Pro (*1)	OSX Sonoma	17	2b0b506	daliborfilus	3m50s	21793/3812	30W?
Intel i7-14700KF	Linux 6.1.0	17	2b0b506	daliborfilus	3m59s	53464/4489	250W
AMD Ryzen 7 7840HS	Linux 6.7.9	17	2b0b506	zw963	4m32s	29045/3799	54W
AMD Ryzen 9 7940HS	Linux 6.11.0	19	675c68c	pfischer	4m37s	30384/3903	54W
AMD Ryzen 7 7735HS	Linux 6.5.0	17	2b0b506	weirdbricks	6m3s	24245/3370	54W
Intel i7-1165G7	Linux 6.6.15	16	9d60848	Xen	6m58s	10304/2826	28W
AMD Ryzen 5 1600	Linux 6.6.10	17	2b0b506	daliborfilus	10m7s	12282/2066	65W
AMD Ryzen 7 3700U	Linux 6.11.0	18	2b0b506	pfischer	10m11s	7449/2025	25W
Intel i7-3615QM (*2)	OSX ?	17	2b0b506	vlazar	10m58s	5223/1714	45W

Notes:

This Apple M1 Pro from daliborfilus was tested on MacPorts and a few workarounds were needed - M1 Pro by self and M1 Max by jgaskins was tested on more common homebrew - maybe it makes some difference in compilation times
MacBook Pro (Retina, 15-inch, Mid 2012)

Nice battle here between top Intels with single core turbo frequency 6GHz and Apple M1 CPUs with frequency 3,2Ghz, pretty impressive.
On OSX, homebrew is frendlier than MacPorts for building Crystal.
Top Intel CPUs are also pretty good heaters (14900KS has 320W TDP - MHz wars ended)

Blacksmoke16 · March 14, 2024, 8:31pm

I moved these comments to its own thread given the other has already been solved.

Also its minor, but you can simplify steps 1 and 4 to just crystal clear_cache and make clean respectively.

Here are my results:

$ time make release=1
Using /usr/bin/llvm-config [version=17.0.6]
clang++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/include -std=c++17   -fno-exceptions -funwind-tables -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="2b0b50602" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1710327137"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -D preview_overload_order --release -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2

real    3m19.817s
user    3m18.175s
sys     0m1.366s

$ neofetch
                   -`                    george@theStone
                  .o+`                   ---------------
                 `ooo/                   OS: Arch Linux x86_64
                `+oooo:                  Kernel: 6.7.9-arch1-1
               `+oooooo:                 Packages: 730 (pacman)
               -+oooooo+:                Shell: bash 5.2.26
             `/:-:++oooo+:               Resolution: 2560x1440
            `/++++/+++++++:              WM: sway
           `/++++++++++++++:             Bar: waybar
          `/+++ooooooooooooo/`           Theme: Breeze-Dark [GTK3]
         ./ooosssso++osssssso+`          Icons: Breeze-Dark [GTK3]
        .oossssso-````/ossssss+`         Terminal: WezTerm
       -osssssso.      :ssssssso.        CPU: Intel i9-14900K (32) @ 5.7GHz
      :osssssss/        osssso+++.       GPU: AMD ATI Radeon RX 7900 XT/7900 XTX/7900M
     /ossssssss/        +ssssooo/-       Memory: 6.96GiB / 62.61GiB (11%)
   `/ossssso+/:-        -:/+osssso+-     Font: Roboto Mono 12 [GTK3]
  `+sso+:-`                 `.-/+oso:    Locale: en_US.UTF-8
 `++:.                           `-/+/
 .`                                 `/

straight-shoota · March 15, 2024, 10:37am

Wow, you got a nice beast there @Blacksmoke16

Blacksmoke16 · March 15, 2024, 1:06pm

Haha yea, recently re-built my setup a few months ago. Deff works quite well yea . But of course https://www.intel.com/content/www/us/en/newsroom/news/intel-core-14th-gen-i9-14900ks-now-available.html comes out a few months later, so maybe if I waited it would be under 3min .

jgaskins · March 15, 2024, 6:58pm

Looks like an M1 Max vs M1 Pro makes a difference here:

➜  crystal git:(master) crystal clear_cache && make clean && time (make crystal release=1)
rm -rf .build
rm -rf ./docs
rm -rf src/llvm/ext/llvm_ext.o
rm -rf man/*.gz
Using /opt/homebrew/Cellar/llvm/17.0.6_1/bin/llvm-config [version=17.0.6]
c++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/opt/homebrew/Cellar/llvm/17.0.6_1/include -std=c++17 -stdlib=libc++   -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="2b0b50602" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1710327137"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -D preview_overload_order --release -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2
( make crystal release=1; )  186.26s user 2.33s system 99% cpu 3:10.22 total

I’d really like to see how fast the M3 Max would compile it, tbh. It has 30% higher single-core benchmarks on Geekbench.

vlazar · March 19, 2024, 5:39pm

Here is 12 years old MacBook Pro (Retina, 15-inch, Mid 2012)

% crystal clear_cache && make clean && time make release=1
Using /usr/local/opt/llvm/bin/llvm-config [version=17.0.6]
c++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/local/Cellar/llvm/17.0.6_1/include -std=c++17 -stdlib=libc++   -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="2b0b50602" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1710327137"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -D preview_overload_order --release -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2
make release=1  640.65s user 7.11s system 98% cpu 10:58.00 total

% sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-3615QM CPU @ 2.30GHz

weirdbricks · March 20, 2024, 1:55am

From within a docker container:

crystal clear_cache && make clean && time make release=1
rm -rf .build
rm -rf ./docs
rm -rf src/llvm/ext/llvm_ext.o
rm -rf man/*.gz
Using /usr/bin/llvm-config [version=17.0.6]
g++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/lib/llvm-17/include -std=c++17   -fno-exceptions -funwind-tables -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="2b0b50602" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1710327137" CC="cc -fuse-ld=lld" CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -D preview_overload_order --release -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2
347.41user 15.87system 6:03.34elapsed 99%CPU (0avgtext+0avgdata 4547248maxresident)k
384inputs+276632outputs (289major+3983477minor)pagefaults 0swaps

neofetch
            .-/+oossssoo+/-.               root@3e3c045d533d 
        `:+ssssssssssssssssss+:`           ----------------- 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu Noble Numbat (development branch) x86_64 
    .ossssssssssssssssssdMMMNysssso.       Kernel: 6.5.0-0.deb12.4-amd64 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Uptime: 20 days, 1 hour, 15 mins 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Packages: 428 (dpkg) 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Shell: bash 5.2.21 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Resolution: 1920x1080 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   CPU: AMD Ryzen 7 7735HS with Radeon Graphics (16) @ 4.829GHz 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   GPU: AMD ATI Radeon 680M 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Memory: 3580MiB / 15265MiB 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+
.ssssssssdMMMNhsssssssssshNMMMdssssssss.                           
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/                            
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

self · March 21, 2024, 11:23pm

git co 2b0b50602 && crystal clear_cache && make clean && time LLVM_CONFIG=/opt/homebrew/Cellar/llvm/17.0.6_1/bin/llvm-config make release=1
# =>  190,60s user 4,05s system 99% cpu 3:15,66 total

Apple M1 Pro, basic model

Xen · March 23, 2024, 8:38pm

OK, had to try on my 3 year old Dell XPS 13 then.

~/d/c/crystal ▶ crystal clear_cache && make clean && time make release=1
rm -rf .build
rm -rf ./docs
rm -rf src/llvm/ext/llvm_ext.o
rm -rf man/*.gz
Using /usr/bin/llvm-config [version= 16.0.6]
g++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/lib/llvm-16/include -std=c++17   -fno-exceptions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="9d6084836" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1705938688"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -D preview_overload_order --release -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2

________________________________________________________
Executed in  418.46 secs    fish           external
   usr time  405.14 secs  233.00 micros  405.14 secs
   sys time   12.85 secs   42.00 micros   12.85 secs

~/d/c/crystal ▶

       _,met$$$$$gg.          xen@xiphos 
    ,g$$$$$$$$$$$$$$$P.       ---------- 
  ,g$$P"     """Y$$.".        OS: Debian GNU/Linux trixie/sid x86_64 
 ,$$P'              `$$$.     Host: XPS 13 9310 
',$$P       ,ggs.     `$$b:   Kernel: 6.6.15-amd64 
`d$$'     ,$P"'   .    $$$    Uptime: 1 day, 4 hours, 6 mins 
 $$P      d$'     ,    $$P    Packages: 2630 (dpkg) 
 $$:      $$.   -    ,d$$'    Shell: fish 3.7.0 
 $$;      Y$b._   _,d$P'      Resolution: 2560x1440 
 Y$$.    `.`"Y$$$$P"'         DE: GNOME 44.9 
 `$$b      "-.__              WM: Mutter 
  `Y$$                        WM Theme: Adwaita 
   `Y$$.                      Theme: Materia-compact [GTK2/3] 
     `$$b.                    Icons: Adwaita [GTK2/3] 
       `Y$$b.                 Terminal: emacs 
          `"Y$b._             CPU: 11th Gen Intel i7-1165G7 (8) @ 4.700GHz 
              `"""            GPU: Intel TigerLake-LP GT2 [Iris Xe Graphics] 
                              Memory: 21835MiB / 31818MiB

jgaskins · May 1, 2024, 5:44pm

My company just started rolling out Apple M3 Pro laptops. I just started working on a project where I’m using Crystal and noticed that the compilation times for my app were pretty decent. Turns out this machine is just absolutely ridiculous. Compiling the compiler in release mode comes in at 2m 25s — 17s without release. CPU Mark

➜  crystal git:(2b0b50602) crystal clear_cache && make clean && (time make crystal release=1)
rm -rf .build
rm -rf ./docs
rm -rf src/llvm/ext/llvm_ext.o
rm -rf man/*.gz
Using /opt/homebrew/Cellar/llvm/18.1.4/bin/llvm-config [version=18.1.4]
CRYSTAL_CONFIG_BUILD_COMMIT="2b0b50602" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1710327137"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -D preview_overload_order --release --link-flags="-L/opt/homebrew/lib" -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2
make crystal release=1  141.25s user 1.93s system 98% cpu 2:25.54 total

Note that I checked out the version of the compiler we were all benchmarking our compilation times on to try to get an apples-to-apples comparison.

The fact that this is ~24% faster than the current top of the list (a 2.5-year-old CPU!) really makes me want to get one for my personal laptop, too, but we’re 6 months into the M3 Pro (released Nov 7, 2023), so I’ll probably wait and get the M4 Pro/Max when it comes out instead.

zw963 · May 3, 2024, 5:16am

Wow, Apple M3 Pro beat @Blacksmoke16 's Intel i9-14900K a lot, unimaginable!

The time savings brought by good machines are too important for Crystal.

adamcarr · August 21, 2024, 11:27pm

I was looking for data on cpu compile times and ran across this. Never used crystal before, but I did a build with an M3 Max and thought I would share the results.

crystal % crystal clear_cache && make clean && (time make crystal release=1)
rm -rf .build
rm -rf ./docs
rm -rf src/llvm/ext/llvm_ext.o
rm -rf man/*.gz
Using /opt/homebrew/Cellar/llvm/18.1.8/bin/llvm-config [version=18.1.8]
CRYSTAL_CONFIG_BUILD_COMMIT="2b0b50602" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1710327137"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -D preview_overload_order --release -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2
ld: warning: reexported library with install name '/opt/homebrew/opt/llvm/lib/libunwind.1.dylib' found at '/opt/homebrew/Cellar/llvm/18.1.8/lib/libunwind.1.0.dylib' couldn't be matched with any parent library and will be linked directly
make crystal release=1  137.36s user 4.68s system 99% cpu 2:22.15 total

Slightly better than M3 Pro, but not by much.

M3 Max (4 Efficiency Cores, 12 Performance Cores), 48GB Memory, Sonoma 14.6.1

jgaskins · November 8, 2024, 11:00pm

Just checked against the latest master branch commit on my M1 Max using Crystal 1.14 and it looks like compiling the compiler has gotten faster in general. It’s now coming in at 2m46s on my M1 Max — previously 3m10s. So it’s gotten 12.5% faster on the same hardware.

I checked because I started a new role last month and my new M4 Max just arrived today, so of course I wanted to check here. It comes in well under 2 minutes on a fresh pull of the repo:

➜  crystal git:(master) crystal clear_cache && time (make clean crystal release=1)
rm -rf .build
rm -rf ./docs
rm -rf src/llvm/ext/llvm_ext.o
rm -rf man/*.gz
Using /opt/homebrew/opt/llvm/bin/llvm-config [version=19.1.3]
CRYSTAL_CONFIG_BUILD_COMMIT="675c68c1c" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1731065563"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -D preview_overload_order --release -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2
( make clean crystal release=1; )  106.06s user 2.53s system 99% cpu 1:48.76 total

I couldn’t compile the compiler at the commit we’ve been using for the benchmark using the latest release of the compiler on Homebrew (it might’ve been an LLVM issue, I can’t remember), so this was against the latest master branch. Since we were also using a previous version of the compiler to compile the compiler, any thoughts on how to track this moving forward?

zw963 · November 9, 2024, 8:41am

Under two minutes, cool!