[solved] LLVM Problem linking on arm?

Help & Support
1

Hi

I have built crystal.o as usual on my amd64 machine, and transfered the .o to my RaspberryPi.
When I try to link it there as usual, i get this and I have no idea where this is coming from.

pi@raspberrypi:~/crystal $ cc 'crystal.o' -o 'crystal'  -rdynamic  /home/pi/crystal/src/llvm/ext/llvm_ext.o `/usr/bin/llvm-config-6.0 --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre -lm /usr/local/lib/libgc.a -lpthread /home/pi/crystal/src/ext/libcrystal.a -levent -lrt -ldl -L/home/pi/crystal/lib -L/usr/lib -L/usr/local/lib
crystal.o: In function `normalize_triple':
/home/mavu/repo/crystal/src/llvm.cr:87: undefined reference to `LLVMExtNormalizeTargetTriple'
crystal.o: In function `initialize':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:5: undefined reference to `LLVMExtNewDIBuilder'
crystal.o: In function `create_compile_unit':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:9: undefined reference to `LLVMExtDIBuilderCreateCompileUnit'
crystal.o: In function `create_file':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:25: undefined reference to `LLVMExtDIBuilderCreateFile'
crystal.o: In function `create_basic_type':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:13: undefined reference to `LLVMExtDIBuilderCreateBasicType'
crystal.o: In function `get_or_create_type_array':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:17: undefined reference to `LLVMExtDIBuilderGetOrCreateTypeArray'
crystal.o: In function `create_subroutine_type':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:21: undefined reference to `LLVMExtDIBuilderCreateSubroutineType'
crystal.o: In function `create_function':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:34: undefined reference to `LLVMExtDIBuilderCreateFunction'
crystal.o: In function `set_current_debug_location':
/home/mavu/repo/crystal/src/llvm/builder.cr:243: undefined reference to `LLVMExtSetCurrentDebugLocation'
crystal.o: In function `create_lexical_block':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:29: undefined reference to `LLVMExtDIBuilderCreateLexicalBlock'
crystal.o: In function `set_current_debug_location':
/home/mavu/repo/crystal/src/llvm/builder.cr:243: undefined reference to `LLVMExtSetCurrentDebugLocation'
crystal.o: In function `name':
/home/mavu/repo/crystal/src/llvm/basic_block.cr:22: undefined reference to `LLVMExtBasicBlockName'
crystal.o: In function `create_basic_type':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:13: undefined reference to `LLVMExtDIBuilderCreateBasicType'
crystal.o: In function `create_enumerator':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:59: undefined reference to `LLVMExtDIBuilderCreateEnumerator'
crystal.o: In function `get_or_create_array':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:55: undefined reference to `LLVMExtDIBuilderGetOrCreateArray'
crystal.o: In function `create_enumeration_type':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:63: undefined reference to `LLVMExtDIBuilderCreateEnumerationType'
crystal.o: In function `create_pointer_type':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:78: undefined reference to `LLVMExtDIBuilderCreatePointerType'
crystal.o: In function `create_replaceable_composite_type':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:82: undefined reference to `LLVMExtDIBuilderCreateReplaceableCompositeType'
crystal.o: In function `create_member_type':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:73: undefined reference to `LLVMExtDIBuilderCreateMemberType'
crystal.o: In function `create_struct_type':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:68: undefined reference to `LLVMExtDIBuilderCreateStructType'
crystal.o: In function `replace_temporary':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:86: undefined reference to `LLVMExtDIBuilderReplaceTemporary'
crystal.o: In function `create_auto_variable':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:39: undefined reference to `LLVMExtDIBuilderCreateAutoVariable'
crystal.o: In function `create_expression':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:47: undefined reference to `LLVMExtDIBuilderCreateExpression'
crystal.o: In function `insert_declare_at_end':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:51: undefined reference to `LLVMExtDIBuilderInsertDeclareAtEnd'
crystal.o: In function `create_parameter_variable':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:43: undefined reference to `LLVMExtDIBuilderCreateParameterVariable'
crystal.o: In function `end':
/home/mavu/repo/crystal/src/llvm/di_builder.cr:90: undefined reference to `LLVMExtDIBuilderFinalize'
crystal.o: In function `write_bitcode_with_summary_to_file':
/home/mavu/repo/crystal/src/llvm/module.cr:62: undefined reference to `LLVMExtWriteBitcodeWithSummaryToFile'
collect2: error: ld returned 1 exit status

crystal.o should have been built using llvm-6.0 because that is the only version those 2 platforms currently have in common. (still using raspbian stretch on the PI and LLVM-6 is the latest one in that version)

I say should have been built, because ā€œmake crystalā€ shows which LLVM version it is using to build, but when cross-compiling I donā€™t see a way to show which LLVM version it is actually using.

to build crystal.o on amd64:

export LLVM_CONFIG=/usr/bin/llvm-config-6.0
./bin/crystal build src/compiler/crystal.cr --cross-compile --target armv6-unknown-linux-gnueabihf -s -D without_openssl -D without_zlib

linker command:

cc 'crystal.o' -o 'crystal'  -rdynamic  /home/pi/crystal/src/llvm/ext/llvm_ext.o `/usr/bin/llvm-config-6.0 --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre -lm /usr/local/lib/libgc.a -lpthread /home/pi/crystal/src/ext/libcrystal.a -levent -lrt -ldl -L/home/pi/crystal/lib -L/usr/lib -L/usr/local/lib

Any ideas where those errors could be coming from?
I guess some LLVM version mismatch? but then how do I specify the correct LLVM versions when building to cross-compile?

edit: Just to be sure, I removed all LLVM versions except 6.0 on my build machine, and that still does not change anything.

2

was this solved? what was the fix if so?

3

Ok, I didnā€™t find the issue, but I found a way to make it work:

  • Download and unpack official release tarball.
  • point CRYSTAL_LIBRARY_PATH to where you unpacked it.
    export CRYSTAL_LIBRARY_PATH=/home/mavu/crystal_build/crystal-0.33.0-1/share/crystal/src/
  • checkout crystal sources and build the compiler locally with the downloaded one:
    PATH=/home/mavu/crystal_build/crystal-0.33.0-1/bin:$PATH make
  • change CRYSTAL_LIBRARY_PATH to crystal sources (where you just built the compiler):
    export CRYSTAL_LIBRARY_PATH=/home/mavu/repo/crystal/src
  • cross-compile for armv6:
    ./bin/crystal build src/compiler/crystal.cr --cross-compile --target armv6-unknown-linux-gnueabihf -s -D without_openssl -D without_zlib
  • copy resulting crystal.o to RaspberryPi
  • git clone / checkout same version of the compiler on the PI
  • make deps on the PI
  • Change the paths and execute the line on the PI the cross-compile command gave you at the end:
    cc 'crystal.o' -o 'crystal' -rdynamic /home/pi/crystal/src/llvm/ext/llvm_ext.o `/usr/bin/llvm-config-6.0 --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre -lm -lgc -lpthread /home/pi/crystal/src/ext/libcrystal.a -levent -lrt -ldl -L/home/pi/crystal/src -L/usr/lib -L/usr/local/lib

That should give you a working compiler that runs on the raspberryPI.
I donā€™t know if it is really neccessary to first build your own compiler on x86-64 and then use that to cross compile.
It is possible that I did something wrong before I tried this way and it will work without doing that.
This process worked for me on Crystal versions 0.30.0 -> 0.33.0 (just tried it on all of those releases)

4 Likes
4

was typing the solution while you posted :slight_smile:

5

ok, this is great.
I did some more tests and I donā€™t know why, but now I can cross-compile without building a local crystal-compiler or downloading a tar-release.

I seem to have fixed something along the way, and the only things that come to mind are that I explicitly set the CRYSTAL_LIBRARY_PATH to the git-checkout of crystal when cross-compiling, and that I installed libgc-dev (debian buster) somewhere in the middle of all this.

I want to do a ā€œcleanā€ test and see if I can reproduce this on a fresh install and provide a better path to crystal on the PI than this (possibly unnecessarily) way above.

3 Likes
6

Hey,

I just tried your instructions but I couldnā€™t get it work. Specifically, the resulting compiler fails to execute a simple snippet such as print("Hello"). So, can you please confirm if the cross-compiled compiler can execute programs?

FYI, I had jotted down similar steps that had yielded better success on raspbian/buster. Today, I tried revising them for debian/buster Docker image and that led me to your question here. You can find my steps here.

Thanks,

7

With the current version of Crystal (1.8.1), there is an llvm_ext.o that is requested for linking against, but is never built. The Makefile will build it, but does not set the cross-compile flags, even if you set ā€˜targetā€™.

8

For those that come later, it was not clear to me that what I needed to do was build llvm_ext.o (via make deps) on my target platform. I was able to get crystal linked this way.

The interactive mode wonā€™t start, but Iā€™m hoping I have enough of the build mode to get the compiler bootstrapped such that I can publish an aarch64 container where I can build future compiler releases and any updates I might make.

9

Hello, if youā€™re interested in a container image for aarch64, you could take a look to mine: hydrofoil-crystal, which is bootstrapped using existing Alpine Linux Crystal version and then cross-compile the compiler to the target platform (both linux/amd64 and linux/arm64 images are provided with the same tag).

I use it for development on Intel and aarch64 when accessing remote, but also for cross-compilation.

Feel free to inspect at the Dockerfile for the process taken.

Hope it helps! :blush:

10

This looks awesome, but how do you run the same Dockerfile on multiple arch at one pass? It seems all the stages would run on the same arch and there is not any qemu invocation.

11

The trick is using --platform on the FROM conditions to use the image for the build platform (could be amd64, could be arm64): hydrofoil-crystal/Dockerfile at main Ā· luislavena/hydrofoil-crystal Ā· GitHub

FROM --platform=$BUILDPLATFORM alpine:3.17.3 AS stage0

Then you build using TARGETARCH:

make crystal release=1 static=1 target=$TARGETARCH-alpine-linux-musl | tail -1 | tee .build/crystal.sh; \

And last you just do FROM without a platform, which will use the one you provided to the docker build commands.

This only works with BuildKit powered setups (been available since 2020 or so) and uses a multi-stage dockerfile to copy things around.

I described things while experimenting in this pull request.

Since emulating arm64 in GitHub runners was painfully slow, Iā€™m using depot.dev for docker builders, which spins up two machines: a Graviton2 (arm64) and a Intel to build both aarch64 and x86_64, respectively, taking like 10 minutes to build both:

image
image932Ɨ474 75.3 KB

Cheers!

1 Like
12

Iā€™ve got an aarch64 gitlab instance where Iā€™m trying to build it. Iā€™ve got buildkit, but Iā€™m trying to use Debian, rather than Alpine. Just hacking, but Iā€™m getting closer.

I have a somewhat odd error from my perspective with trying to get interactive mode working where a stat reference isnā€™t resolved. Not sure why it cannot link properly with functions in libc!

13

Just for you probably not know.

interactive mode not work with static build for linux currently.

1 Like
14

But the interpreter/interactive mode should work on ARM, correct? Iā€™m no longer trying to build staticā€¦ I just want to start with a working ARM build (with interactive mode) on Debian 11.