A github action template for do release/assets when push a tag. it should fitable for any shards

I push my first hello world crystal shards to github today.

the main purpose for this shard is to write a github action, to build a static version binary, automatically, when i push a tag into github, then i can download from there to use anywhere (linux only)

because i found many famous shards in github never use this automate process, so i share my action file here.

Almost just copy from our gh-actions document, with a little improvement for do release and create a tarball, it good to as a start point anyway.

There is one more question, from [document](Static Linking - Crystal]., i saw this:

With pre-installed crystal compiler, shards, and static libraries of all of stdlib's dependencies these Docker images allow to easily build static Crystal binaries even from glibc-based systems. The official Crystal compiler builds for Linux are created using these images.

so, i consider build static binary directly from normal distro (e.g. i use arch linux) is possible nowaday, right?

The context is, i installed musl packages correctly, i can build several C program binary use make like make LDFLAGS=-static CC=musl-gcc successful, i even can build some rust package e.g. famous fd successful use following command.

$: cargo build --target x86_64-unknown-linux-musl --release

But, i don’t know how to do this with Crystal, there are some discuss several years ago,
maybe it time to open a new thread for discuss this, so, in fact when i try to build on my own distro, i get following error msg:

 ╰─ $ 130  CC=musl-gcc shards build --production --release --progress --static --no-debug --link-flags="-s -Wl,-z,relro,-z,now"
Dependencies are satisfied
Building: crystal_hello_world
Error target crystal_hello_world failed to compile:
/usr/sbin/ld: cannot find -lpcre (this usually means you need to install the development package for libpcre): No such file or directory
/usr/sbin/ld: cannot find -levent (this usually means you need to install the development package for libevent): No such file or directory
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `musl-gcc "${@}" -o /home/common/Study/Crystal/crystal_hello_world/bin/crystal_hello_world -s -Wl,-z,relro,-z,now -rdynamic -static -L/home/zw963/Crystal/bin/../lib/crystal -lpcre -lm -lgc -lpthread -levent  -lrt -lpthread -ldl`

could anyone help on how to resolve this?

Thank you.

4 Likes

I’m not intimately familiar with GitHub Actions, but if you’re building with an Alpine container, the only thing you need to do to get a static binary is add --static. This is an example of the Dockerfile I use to build Kubernetes operators, which gives me 5-6MB container images:

All I’m doing is running shards build --release --static and copying the resulting binary to an empty container image so that the container I deploy only contains the single binary.

2 Likes

All I’m doing is running shards build --release --static and copying the resulting binary to an empty container image so that the container I deploy only contains the single binary.

I guess i know what you means, it similar to docker multi-staging feature, when done build, copy the result to a refresh new container to reduce size.

What i done is similar, but different, i just build a static binary use Alpine when push a tag to github, and let user can download directly from git release page, e.g. it can build a release download link like this:

https://github.com/zw963/crystal_hello_world/releases/download/v0.0.9/crystal_hello_world-v0.0.9-x86_64-unknown-linux-musl.tar.gz

@jgaskins , BTW, It really nice to see familiar face here, i have used your clear water, though, just for play a while, do you think if there is any chance to generate Javascript use Crystal as does with Opal ?

2 Likes

Interesting approach @zw963, I’m currently using my own dev container to build these static binaries:

It uses docker run directly (exp-crystal-hello-binary/build.yml at main · luislavena/exp-crystal-hello-binary · GitHub) since I’m going to publish multi-arch container images soon (both amd64 and arm64).

Then using Docker QEMU support you can docker run --platform linux/arm64 all on the same host.

I’m also working on getting static binaries for macOS (both x86_64 and arm64), that’s a bit challenging as requires the dependencies be statically compiled as well, but not impossible:


The binary is cross-compiled using zig cc and linked against the static libraries generated previously.

(I know, using another language as toolchain, but seems I’m not the only one)

Happy to see others interested in getting the static binaries of their Crystal programs!

5 Likes

Awesome, you build your’s binary from the scratch!

Hope you can share how you build macOS binary from linux! (maybe use VirtualBox)?

And from after visit your’s repo, i learned to use a new good tool, watchexec, thank you!

Hi, i like to research this, where is the zig cc involves? i clone both git project, and search zig, but nothing?

Thank you.

This was a proof of concept and an attempt to get that working on GitHub Actions.

Since then I took a different approach, which I covered in my videos:

And the repository that collects all the static libraries for Linux (Alpine) and macOS is here: GitHub - luislavena/magic-haversack: Facilitate Crystal cross-compilation

Cheers.

3 Likes

And I just found that we had this conversation in the past here! :sweat_smile:

Getting older and forgetting a bunch of things, sorry about that!

Thanks, I haven’t been watch the video yet, and I didn’t think the project name was this.

Hi, when i run rake fetch:all, get following error.

 ╰─ $ rake fetch:all
mkdir -p tmp
mkdir -p downloads/aarch64-linux-musl
mkdir -p lib/aarch64-linux-musl/pkgconfig
Downloading gc-dev-8.2.2-r2.apk (100%) 
mkdir -p tmp/ports/aarch64-linux-musl/gc/8.2.2
tar -tf /home/zw963/Crystal/play/magic-haversack/downloads/aarch64-linux-musl/archives/gc-dev-8.2.2-r2.apk 2>/dev/null | grep -E 'libgc.a|bdw-gc.pc' | xargs -I '{}' tar -xf /home/zw963/Crystal/play/magic-haversack/downloads/aarch64-linux-musl/archives/gc-dev-8.2.2-r2.apk -C tmp/ports/aarch64-linux-musl/gc/8.2.2 --no-anchored '{}' 2>/dev/null
cp tmp/ports/aarch64-linux-musl/gc/8.2.2/usr/lib/libgc.a lib/aarch64-linux-musl/libgc.a
cp tmp/ports/aarch64-linux-musl/gc/8.2.2/usr/lib/pkgconfig/bdw-gc.pc lib/aarch64-linux-musl/pkgconfig/bdw-gc.pc
Downloading gmp-dev-6.2.1-r3.apk (100%) 
mkdir -p tmp/ports/aarch64-linux-musl/gmp/6.2.1
tar -tf /home/zw963/Crystal/play/magic-haversack/downloads/aarch64-linux-musl/archives/gmp-dev-6.2.1-r3.apk 2>/dev/null | grep -E 'libgmp.a|gmp.pc' | xargs -I '{}' tar -xf /home/zw963/Crystal/play/magic-haversack/downloads/aarch64-linux-musl/archives/gmp-dev-6.2.1-r3.apk -C tmp/ports/aarch64-linux-musl/gmp/6.2.1 --no-anchored '{}' 2>/dev/null
cp tmp/ports/aarch64-linux-musl/gmp/6.2.1/usr/lib/libgmp.a lib/aarch64-linux-musl/libgmp.a
cp tmp/ports/aarch64-linux-musl/gmp/6.2.1/usr/lib/pkgconfig/gmp.pc lib/aarch64-linux-musl/pkgconfig/gmp.pc
Downloading libevent-static-2.1.12-r6.apk (100%) 
Downloading libevent-dev-2.1.12-r6.apk (100%) 
mkdir -p tmp/ports/aarch64-linux-musl/libevent/2.1.12
tar -tf /home/zw963/Crystal/play/magic-haversack/downloads/aarch64-linux-musl/archives/libevent-static-2.1.12-r6.apk 2>/dev/null | grep -E 'libevent.a|libevent.pc|libevent_pthreads.a|libevent_pthreads.pc' | xargs -I '{}' tar -xf /home/zw963/Crystal/play/magic-haversack/downloads/aarch64-linux-musl/archives/libevent-static-2.1.12-r6.apk -C tmp/ports/aarch64-linux-musl/libevent/2.1.12 --no-anchored '{}' 2>/dev/null
tar -tf /home/zw963/Crystal/play/magic-haversack/downloads/aarch64-linux-musl/archives/libevent-dev-2.1.12-r6.apk 2>/dev/null | grep -E 'libevent.a|libevent.pc|libevent_pthreads.a|libevent_pthreads.pc' | xargs -I '{}' tar -xf /home/zw963/Crystal/play/magic-haversack/downloads/aarch64-linux-musl/archives/libevent-dev-2.1.12-r6.apk -C tmp/ports/aarch64-linux-musl/libevent/2.1.12 --no-anchored '{}' 2>/dev/null
cp tmp/ports/aarch64-linux-musl/libevent/2.1.12/usr/lib/libevent.a lib/aarch64-linux-musl/libevent.a
cp tmp/ports/aarch64-linux-musl/libevent/2.1.12/usr/lib/libevent_pthreads.a lib/aarch64-linux-musl/libevent_pthreads.a
cp tmp/ports/aarch64-linux-musl/libevent/2.1.12/usr/lib/pkgconfig/libevent.pc lib/aarch64-linux-musl/pkgconfig/libevent.pc
cp tmp/ports/aarch64-linux-musl/libevent/2.1.12/usr/lib/pkgconfig/libevent_pthreads.pc lib/aarch64-linux-musl/pkgconfig/libevent_pthreads.pc
2 retrie(s) left for openssl-libs-static-3.1.3-r0.apk (404 Not Found)
1 retrie(s) left for openssl-libs-static-3.1.3-r0.apk (404 Not Found)
0 retrie(s) left for openssl-libs-static-3.1.3-r0.apk (404 Not Found)
404 Not Found
rake aborted!
Errno::ENOENT: No such file or directory @ rb_sysopen - downloads/aarch64-linux-musl/archives/openssl-libs-static-3.1.3-r0.apk
/home/common/.rvm/gems/ruby-3.2.1/gems/mini_portile2-2.8.1/lib/mini_portile2/mini_portile.rb:335:in `verify_file'
/home/common/.rvm/gems/ruby-3.2.1/gems/mini_portile2-2.8.1/lib/mini_portile2/mini_portile.rb:83:in `block in download'
/home/common/.rvm/gems/ruby-3.2.1/gems/mini_portile2-2.8.1/lib/mini_portile2/mini_portile.rb:81:in `each'
/home/common/.rvm/gems/ruby-3.2.1/gems/mini_portile2-2.8.1/lib/mini_portile2/mini_portile.rb:81:in `download'
/home/zw963/Crystal/play/magic-haversack/Rakefile:71:in `block (3 levels) in <top (required)>'
Tasks: TOP => fetch:all => fetch:aarch64-linux-musl => fetch:aarch64-linux-musl:openssl
(See full trace by running task with --trace)

with trace
 ╰─ $ 1  rake fetch:all --trace
** Invoke fetch:all (first_time)
** Invoke fetch:aarch64-linux-musl (first_time)
** Invoke fetch:aarch64-linux-musl:gc (first_time)
** Invoke tmp (first_time, not_needed)
** Invoke downloads/aarch64-linux-musl (first_time, not_needed)
** Invoke lib/aarch64-linux-musl/pkgconfig (first_time, not_needed)
** Execute fetch:aarch64-linux-musl:gc
** Invoke fetch:aarch64-linux-musl:gmp (first_time)
** Invoke tmp (not_needed)
** Invoke downloads/aarch64-linux-musl (not_needed)
** Invoke lib/aarch64-linux-musl/pkgconfig (not_needed)
** Execute fetch:aarch64-linux-musl:gmp
** Invoke fetch:aarch64-linux-musl:libevent (first_time)
** Invoke tmp (not_needed)
** Invoke downloads/aarch64-linux-musl (not_needed)
** Invoke lib/aarch64-linux-musl/pkgconfig (not_needed)
** Execute fetch:aarch64-linux-musl:libevent
** Invoke fetch:aarch64-linux-musl:openssl (first_time)
** Invoke tmp (not_needed)
** Invoke downloads/aarch64-linux-musl (not_needed)
** Invoke lib/aarch64-linux-musl/pkgconfig (not_needed)
** Execute fetch:aarch64-linux-musl:openssl
2 retrie(s) left for openssl-libs-static-3.1.3-r0.apk (404 Not Found)
1 retrie(s) left for openssl-libs-static-3.1.3-r0.apk (404 Not Found)
0 retrie(s) left for openssl-libs-static-3.1.3-r0.apk (404 Not Found)
404 Not Found
rake aborted!
Errno::ENOENT: No such file or directory @ rb_sysopen - downloads/aarch64-linux-musl/archives/openssl-libs-static-3.1.3-r0.apk
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/3.2.0/digest.rb:64:in `initialize'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/3.2.0/digest.rb:64:in `open'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/3.2.0/digest.rb:64:in `file'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/3.2.0/digest.rb:49:in `file'
/home/common/.rvm/gems/ruby-3.2.1/gems/mini_portile2-2.8.1/lib/mini_portile2/mini_portile.rb:335:in `verify_file'
/home/common/.rvm/gems/ruby-3.2.1/gems/mini_portile2-2.8.1/lib/mini_portile2/mini_portile.rb:83:in `block in download'
/home/common/.rvm/gems/ruby-3.2.1/gems/mini_portile2-2.8.1/lib/mini_portile2/mini_portile.rb:81:in `each'
/home/common/.rvm/gems/ruby-3.2.1/gems/mini_portile2-2.8.1/lib/mini_portile2/mini_portile.rb:81:in `download'
/home/zw963/Crystal/play/magic-haversack/Rakefile:71:in `block (3 levels) in <top (required)>'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/home/common/.rvm/rubies/ruby-3.2.1/lib/ruby/gems/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/home/zw963/.rvm/rubies/ruby-3.2.1/bin/rake:25:in `load'
/home/zw963/.rvm/rubies/ruby-3.2.1/bin/rake:25:in `<main>'
Tasks: TOP => fetch:all => fetch:aarch64-linux-musl => fetch:aarch64-linux-musl:openssl

@luislavena, another concern.

Does fixuid is useful in this case? i consider fixuid only make sense if you mount code into container directly instead COPY, following code should work in most cases.

RUN addgroup -g 1000 docker && \
    adduser -u 1000 -G docker -h /home/docker -s /bin/sh -D docker

COPY --chown=docker:docker src src

@zw963 please be careful, you’re looking at something that was done as proof of concept, unmaintained and not updated in 3 months, which are ages in computer terms.

The error there is clear:

2 retrie(s) left for openssl-libs-static-3.1.3-r0.apk (404 Not Found)
1 retrie(s) left for openssl-libs-static-3.1.3-r0.apk (404 Not Found)
0 retrie(s) left for openssl-libs-static-3.1.3-r0.apk (404 Not Found)
404 Not Found

Is not able to find the .apk file, most likely there is a new version of openssl and these files are gone from the CDN.

Perhaps I’m missing something in your comment, would you mind elaborate further?

In my particular case. the usage of fixuid is to run the container as the current user (Eg. docker run -u $(id -u):$(id -g) so the artifacts generated by the build process (when running the container) are owned by the current user, which might be different than 1000 as your example. FIXUID corrects that.

I’m not copying src inside the container or changing the ownership as the container image I want needs to be reusable (I cannot be rebuilding the image every time I change the source code).

Cheers.

Alright, if this is your’s concern, that ok. :smile:

For me, just used by myself, user/group id always 1000, the concerns is file permissions, maybe i should add fixuid back. :joy:

you’re looking at something that was done as proof of concept, unmaintained and not updated in 3 months, which are ages in computer terms.

Okay, i will check that.

BTW, instead of fixuid the hard way (add -u when run, add many code to initialize fixuid), i probably prefer to write a user wrapper script to run it.

sed "s#addgroup -g 1000 docker#addgroup -g $(id -g) docker#" $dockerfile > $tmp_dockerfile
sed -i "s#adduser -u 1000 -G docker#adduser -u $(id -u) -G docker#" $tmp_dockerfile

docker build ... -f $tmp_dockerfile

Whatever rocks your boat. The code is out there, feel free to adapt to your needs.

Cheers.

1 Like