Segfault while bootstraping from ruby

GeopJr · October 10, 2023, 4:56pm

I’ve been trying to bootstrap the compiler from ruby using GitHub - crystal-lang/bootstrap-script: Automated script to bootstrap the crystal compiler from source, with the goal of updating the script to the latest versions & using containers, but can’t get past the second stage:

==> Bootstrapping Crystal (001/160)
patching file bin/crystal
patching file bootstrap/crystal/compiler.cr
patching file bootstrap/crystal/program.cr
patching file lib/crystal/compiler.rb
                           user     system      total        real
parse                  0.000000   0.000000   0.000000 (  0.000615)
normalize              2.910000   0.120000   3.040000 (  3.030746)
type inference:       26.740000   1.190000  27.930000 ( 27.953474)
fix empty types        0.350000   0.000000   0.350000 (  0.352627)
afert type inference   0.730000   0.020000   0.750000 (  0.739854)
codegen-llvm          17.450000   0.930000  18.380000 ( 18.401006)
codegen-llc          .crystal//home/geopjr/bootstrap-script/buildroot/src/crystal/bootstrap/crystal.cr/main.s: Assembler messages:
.crystal//home/geopjr/bootstrap-script/buildroot/src/crystal/bootstrap/crystal.cr/main.s:8847: Warning: stand-alone `data16' prefix
.crystal//home/geopjr/bootstrap-script/buildroot/src/crystal/bootstrap/crystal.cr/main.s:8849: Warning: stand-alone `data16' prefix
.crystal//home/geopjr/bootstrap-script/buildroot/src/crystal/bootstrap/crystal.cr/main.s:8850: Warning: stand-alone `data16' prefix
  0.850000   0.170000  54.040000 ( 54.169867)
codegen-clang        /usr/bin/ld: .crystal//home/geopjr/bootstrap-script/buildroot/src/crystal/bootstrap/crystal.cr/main.o: in function `_2A_Crystal_3A__3A_Compiler_23_compile_3C_Crystal_3A__3A_Compiler_3E__3A_Nil':
.crystal//home/geopjr/bootstrap-script/buildroot/src/crystal/bootstrap/crystal.cr/main.bc:(.text+0x11b46): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
/usr/bin/ld: warning: .crystal//home/geopjr/bootstrap-script/buildroot/src/crystal/bootstrap/crystal.cr/main.o: missing .note.GNU-stack section implies executable stack
/usr/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
  0.000000   0.110000   0.570000 (  0.576038)
TOTAL:                49.030000   2.540000 105.060000 (105.224228)
==> Bootstrapping Crystal (002/160)
Normalize: 0.618294 seconds
Type inference: 10.2995 seconds
Codegen (crystal): 5.31784 seconds
Codegen (bitcode): 1.07611 seconds
./bootstrap: line 192: 24294 Segmentation fault      (core dumped) "$input" "${crystal_args[@]}" -o "$output" src/compiler/crystal.cr
Normalize: 0.598149 seconds
Type inference: 10.868 seconds
Codegen (crystal): 5.3217 seconds
Codegen (bitcode): 1.17425 seconds
./bootstrap: line 192: 25957 Segmentation fault      (core dumped) "$input" "${crystal_args[@]}" -o "$output" src/compiler/crystal.cr

Using the compiler itself is a bit more verbose:

1-crystal -e "puts \"foo\""

No such file or directory:
_2A_Exception_23_initialize_3C_Errno_2C__20_String_3E__3A_Array_28_String_29_ + [0]
_2A_Errno_23_initialize_3C_Errno_3E__3A_Array_28_String_29_ + [0]
_2A_Errno_3A__3A_new_3C_Errno_3A_Class_3E__3A_Errno + [0]
_2A_File_3A__3A_expand_path_3C_File_3A_Class_2C__20_String_3E__3A_String + [0]
_2A_Crystal_3A__3A_Program_23_require_from_load_path_3C_Crystal_3A__3A_Program_2C__20_String_3E__3A_Nil_20__7C__20_Crystal_3A__3A_ASTNode_2B_ + [0]
_2A_Crystal_3A__3A_Program_23_require_3C_Crystal_3A__3A_Program_2C__20_String_2C__20_Nil_20__7C__20_String_20__7C__20_Crystal_3A__3A_VirtualFile_3E__3A_Nil_20__7C__20_Crystal_3A__3A_ASTNode_2B_ + [0]
_2A_Crystal_3A__3A_Normalizer_23_transform_3C_Crystal_3A__3A_Normalizer_2C__20_Crystal_3A__3A_Require_2B__3E__3A_Crystal_3A__3A_ASTNode_2B_ + [0]
_2A_Crystal_3A__3A_ASTNode_2B__23_transform_3C_Crystal_3A__3A_ASTNode_2B__2C__20_Crystal_3A__3A_Normalizer_3E__3A_Crystal_3A__3A_ASTNode_2B_ + [0]
_2A_Crystal_3A__3A_Normalizer_23_transform_3C_Crystal_3A__3A_Normalizer_2C__20_Crystal_3A__3A_Expressions_2B__3E__3A_Crystal_3A__3A_ASTNode_2B_ + [0]
_2A_Crystal_3A__3A_ASTNode_2B__23_transform_3C_Crystal_3A__3A_ASTNode_2B__2C__20_Crystal_3A__3A_Normalizer_3E__3A_Crystal_3A__3A_ASTNode_2B_ + [0]
_2A_Crystal_3A__3A_Normalizer_23_normalize_3C_Crystal_3A__3A_Normalizer_2C__20_Crystal_3A__3A_ASTNode_2B__3E__3A_Crystal_3A__3A_ASTNode_2B_ + [0]
_2A_Crystal_3A__3A_Program_23_normalize_3C_Crystal_3A__3A_Program_2C__20_Crystal_3A__3A_ASTNode_2B__3E__3A_Crystal_3A__3A_ASTNode_2B_ + [0]
_2A_Crystal_3A__3A_Compiler_23_compile_3C_Crystal_3A__3A_Compiler_3E__3A_Nil + [0]
__crystal_main + [0]
main + [0]
__libc_init_first + [0]
__libc_start_main + [0]
_start + [0]

More info:

if you attempt to run the script, it will fail due to pcl’s download location

-- curl -L -o "$downloads"/pcl-1.12.tar.gz http://xmailserver.org/pcl-1.12.tar.gz
++ curl -L -o "$downloads"/pcl-1.12.tar.gz http://www.xmailserver.org/pcl-1.12.tar.gz

LLVM 3 requires python 2 (?) which might have been replaced by 3 on your distro (and removed completely from its repos), you can run the following to install it in the build env (run it when it fails and start again after it finishes):

curl -L -o ./downloads/Python-2.7.3.tgz http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz && cd ./buildroot/src/ && rm -Rf Python-2.7.3.tgz && tar xzf ../../downloads/Python-2.7.3.tgz && cd Python-2.7.3 && ./configure --prefix="$(pwd)"/../../ --enable-shared && make && make install && cd ../../../ && rm -rf buildroot/src/llvm-3.3.src/

It probably won’t be able to find librt and you might need to symlink it (not sure if this is the best approach but I didn’t want to modify the stage patches yet): ln -s /usr/lib/x86_64-linux-gnu/librt.so.1 ./buildroot/lib/librt.so
The script fails on my machine during linking one of the deps, so all my attempts have been on debian containers and vms, including debian versions before the glibc librt & python change, so I doubt the symlink or python2 is the issue

Any ideas?

beta-ziliani · October 11, 2023, 9:22pm

what’s the goal?

beta-ziliani · October 11, 2023, 9:23pm

I mean, you mention updating the script and using the container, but do you have a more specific goal?

GeopJr · October 11, 2023, 10:11pm

do you have a more specific goal?

Yes, kind-of… there’s a certain distro in alpha with an unusual stack and while figuring out what would be the best way to package Crystal for it I went down the bootstrapping from Ruby rabbit hole. Fixing the script is not that important for reaching that goal as there are better ways to port it over but it’d be nice to have it as a working alternative.

However, I’m about to give up as even using the packages of the time yields the same result and I don’t think anyone is up to debugging a decade old build:

# git clone https://github.com/crystal-lang/crystal/ -b ruby
FROM ubuntu:precise
WORKDIR /app
COPY . .

RUN cat llvm-3.3.tar.gz | tar xz --strip-components=1 -C /usr
RUN sed -i -re 's/([a-z]{2}\.)?archive.ubuntu.com|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list
RUN apt-get update && apt-get install -y  zlib1g-dev ruby1.9.3 build-essential git libpcre3-dev libunwind7-dev wget


WORKDIR /app/crystal
RUN gem install bundler -v '< 1.8' --no-ri --no-rdoc
RUN bundle
# RUN bin/crystal -e "puts 1" # Works
RUN bin/crystal -stats bootstrap/crystal.cr
RUN ./crystal -e "puts 1"
# segfault

tycode · November 18, 2024, 2:54pm

Hi, @GeopJr and other readers…

I’ve also tried to get the bootstrap script working and ran into similar issues (on Debian 12 “bookworm”, in my case). Here’s what I did so far:

Like you, I saw I needed Python 2. I built Python 2.7.18 from source and made a symlink so that the python2 command would run this version.
I saw a message about needing to install GraphViz. This is a red herring: GraphViz was already installed, and this message is displayed regardless of whether or not it’s installed. So, you don’t need to worry about this message.
I had an error about “Cannot find llvm-config”. This was displayed immediately after the GraphViz message (which is why I ended up looking into that). I’m still not sure why this happened, but after a bit of investigating, I ended up just doing rm -rf buildroot/src/llvm-3.3.src and re-running the bootstrap script, and it worked that time.
I got the error with librt.so. Now I can share some useful information about this error: it happens because that library has been removed from modern glibc. It was only needed by time.linux.cr, which is easily fixed with the following patch (add this to the end of stage1.patch, and note that the empty-looking line before class Time needs a single space on it to be a valid patch file):

diff --git a/std/time.linux.cr b/std/time.linux.cr
index 8aa07734b..dbe094b9a 100644
--- a/std/time.linux.cr
+++ b/std/time.linux.cr
@@ -1,4 +1,4 @@
-lib Librt("rt")
+lib C
   struct TimeSpec
     tv_sec, tv_nsec : Int64
   end
@@ -7,7 +7,7 @@ end
 
 class Time
   def initialize
-    Librt.clock_gettime(0, out time)
+    C.clock_gettime(0, out time)
     @seconds = time.tv_sec + time.tv_nsec / 1e9
   end
 end

If you search for clock_gettime, you might find yourself [here]( https:// www . man7 . org /linux/man-pages/man3/clock_gettime.3.html ), which states “Link with -lrt (only for glibc versions before 2.17)”; that’s how I figured to just change Librt to C.

I got an error about libunwind.so. I tried the following workaround (I don’t suggest doing this, as it doesn’t seem to work, see below):

# as root
cd /usr/lib/x86_64-linux-gnu
ln -s libunwind.so.8.0.1 libunwind.so

Doing this does allow stage1 to build, but when it tries to actually run it, it gets a segfault - maybe the same segfault that you’re getting…

I did do a little research, which reveals there’s multiple different libunwind projects, and maybe we’re linking to the wrong one. My libunwind.so.8.0.1, which is part of my Debian 12 system, comes from [“nongnu” libunwind]( http:// www . nongnu . org /libunwind ), whatever “nongnu” means. But there’s an LLVM libunwind as well, as discussed [here]( https:// github . com /libunwind/libunwind/discussions/665 ), and I see it’s mentioned [here]( https:// clang . llvm . org /docs/Toolchain.html#libunwind-llvm ) in the LLVM docs. Since Crystal uses LLVM I’m pondering whether I need to somehow get it to link with the LLVM libunwind instead of the “nongnu” one, and whether that’ll resolve the segfault.

But I’m getting very out of my depth here, so that’s where I stopped. If anyone has some pointers for how to make further progress, I’d really appreciate it!

Hopefully related, I’ve also posted [over on the Debian User Forums]( https:// forums . debian . net /viewtopic.php?t=160984 ) to try to find out how Debian is compiling Crystal, since I feel sure they must be bootstrapping it somehow.

PS. This “new users can only put two links in a post” rule isn’t very nice! I’ve bloated all the links with spaces to get round this, you’re just going to have to manually de-space them.