Issue with Running Crystal Executable on Another macOS Machine

galapagdos · August 8, 2024, 11:04pm

I’m currently working on a Crystal project, and I’ve run into an issue when trying to run my compiled binary on another macOS machine. The application throws the following error:

dyld[29586]: Library not loaded: /usr/local/opt/pcre2/lib/libpcre2-8.0.dylib
  Referenced from: <2270A1BE-CF50-31DF-8947-32B67C9A0FCD> /Users/SOMEONE/Downloads/SOMEExECUTABLE
  Reason: tried: '/usr/local/opt/pcre2/lib/libpcre2-8.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/pcre2/lib/libpcre2-8.0.dylib' (no such file), '/usr/local/opt/pcre2/lib/libpcre2-8.0.dylib' (no such file)
zsh: abort      ./SOMEPROGRAM

The issue arises because the target machine does not have libpcre2-8.0.dylib installed in the expected location.
I compiled the program with the following command to link the static library:

crystal build your_program.cr --release --link-flags="/usr/local/opt/pcre2/lib/libpcre2-8.a"

However, it seems like the binary still requires the dynamic library at runtime.

My questions are:

How can I fully statically link libpcre2 and other dependencies so that the binary does not require the dynamic library at runtime?
Is there a way to compile my Crystal program to avoid needing pcre2 altogether?
Are there best practices for creating a fully portable Crystal binary on macOS?

Any guidance on how to solve this issue would be greatly appreciated!

Thank you!

kojix2 · August 9, 2024, 12:45am

As you may already know, the official Crystal documentation says

https://crystal-lang.org/reference/1.13/guides/static_linking.html#macos

macOS¶

macOS doesn’t officially support fully static linking because the required system libraries are not available as static libraries.

The solutions for this issue are

Using the install_name_tool command to modify the path of the dylib.
Preparing a Homebrew tap for your tool.
Creating an App Bundle.

Many Mac users manage their tools with Homebrew, so option 2 is simple.
I haven’t tried option 3, so I hope someone more experienced can explain how to do it.

MyApp.app/
└── Contents/
    ├── MacOS/
    │   ├── my_executable           # Crystal executable file
    │   └── launch.sh               # Shell script
    ├── Frameworks/
    │   └── libpcre2-8.0.dylib      # Required library
    └── Info.plist                  # Metadata file

straight-shoota · August 9, 2024, 5:20am

macOS doesn’t support fully statically linked binaries, but the missing part is the system library. It should be entirely possible to statically link libpcre2.

Following the instructions on Static Linking - Crystal. You’ll need a static build of libpcre2 for that.

libpcre2 is used for regular expressions. If your program doesn’t use regular expressions, you wouldn’t need libpcre2.

Fully portable builds are impossible on macOS because there is no static version of the system library.
So portability only reaches as far as the system library is binary compatible with other versions of the operating system.
So there’s some limit.

I don’t have much experience with building for macOS, maybe others can chip in on that.

aboulafia · August 10, 2024, 4:07pm

For what it’s worth I don’t seem to have problems with static builds on macOS

Dynamic build :

crystal build my_toy_lang.cr
otool -L my_toy_lang
/opt/local/lib/libpcre2-8.0.dylib (compatibility version 14.0.0, current version 14.0.0)|
/opt/local/lib/libgc.1.dylib (compatibility version 7.0.0, current version 7.3.0)|
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)|
/opt/local/lib/libevent-2.1.7.dylib (compatibility version 8.0.0, current version 8.1.0)|
/opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)|

Static build :

crystal build my_toy_lang.cr --static
otool -L my_toy_lang
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)

Only the system lib is still dynamic but not the other ones (and obviously the size of the executable increases with static build)

galapagdos · August 10, 2024, 8:46pm

I din’t need regex at all, even basic program as:
puts “Hello World!” is using it as I can see in otool:

otool -L ./helloworld      
./helloworld:
        /opt/homebrew/opt/pcre2/lib/libpcre2-8.0.dylib (compatibility version 14.0.0, current version 14.0.0)

Is there’s a way to build it without this library?

galapagdos · August 10, 2024, 8:48pm

encountered an issue while trying to create static builds on macOS. When running the following command:

bash

crystal build helloworld.cr --static

I received the following error:

ld: library not found for -lcrt0.o (this usually means you need to install the development package for libcrt0.o)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with exit status 1: cc "${@}" -o /Users/idan/Documents/dictionary/helloworld -rdynamic -static -L/opt/homebrew/Cellar/crystal/1.13.1/bin/../../../../lib -L/opt/homebrew/Cellar/pcre2/10.44/lib -lpcre2-8 -D_THREAD_SAFE -pthread -L/opt/homebrew/Cellar/bdw-gc/8.2.6/lib -lgc -lpthread -L/opt/homebrew/Cellar/libevent/2.1.12_1/lib -levent -liconv

It seems that the linker is unable to find -lcrt0.o, which is typically indicative of a missing development package. I would appreciate any advice on how to resolve this issue and successfully build a static binary.

Thank you very much for your help!

kojix2 · August 11, 2024, 10:10am

The discussion here involves two different topics.

(1) Creating a statically linked executable on Mac

I think this is very challenging. Many sites like StackOverflow appear when searching for issues related to missing -lcrt0, but there doesn’t seem to be an easy solution.

(2) Do not link libpcre2 when regular expressions are not used

This issue is different from (1), and I felt it was worth investigating further.

hw1.cr

puts "hello world"

At first glance, this code seems simple, but it’s not as straightforward as it looks. It uses strings and performs the complex operation of outputting to standard output. Let’s compile this code and see what functions are included.

crystal build --release hw1.cr
nm hw1

You’ll notice that it includes many functions. Unless specified otherwise, Crystal loads the following libraries:

github.com

crystal-lang/crystal/blob/master/src/prelude.cr

# Entries to this file should only be ordered if macros are involved -
# macros need to be defined before they are used.
# A first compiler pass gathers all classes and methods, removing the
# requirement to place these in load order.
#
# When adding new files, use alpha-sort when possible. Make sure
# to also add them to `docs_main.cr` if their content needs to
# appear in the API docs.

# This list requires ordered statements
require "crystal/once"
require "lib_c"
require "macros"
require "object"
require "comparable"
require "exception"
require "iterable"
require "iterator"
require "steppable"
require "indexable"

This file has been truncated. show original

For much simpler code, the standard library isn’t necessary.

a = 3

crystal build --prelude empty --release simple.cr
nm simple

Here, the number of functions is significantly reduced.

nm hw1 | grep pcre2

                 U pcre2_config_8

You can see that this function is being called.

Crystal can output intermediate representation llvm-ir.

crystal build --emit llvm-ir hw1.cr

hw1.ll can be compiled like this:

clang hw1.ll -lm -lz -levent -lgc -lpcre2-8

Let’s open hw1.ll in a text editor and search for pcre2. You’ll find the following declaration:

declare i32 @pcre2_config_8(i32, ptr) local_unnamed_addr

It’s only used here:

  %41 = call i32 @pcre2_config_8(i32 11, ptr %40), !dbg !17427

Let’s try modifying the following line like this. It will likely fail since ptr %40.

  %41 = add i32 0, 17

Let’s try compiling without pcre2.

clang hw1.ll -lm -lz -levent -lgc

We did it. It compiles without pcre2.

ldd a.out

Indeed, it’s not linked.

Running it produces an error.

Unhandled exception: Invalid libpcre2 version (RuntimeError)
  from /usr/local/share/crystal/src/regex/pcre2.cr:18:33 in '~Regex::PCRE2::version_number:init'

The issue lies here.

module Regex::PCRE2
  @re : LibPCRE2::Code*
  @jit : Bool

  def self.version : String
    String.new(24) do |pointer|
      size = LibPCRE2.config(LibPCRE2::CONFIG_VERSION, pointer) ## %41 HERE!!
      {size - 1, size - 1}
    end
  end

  class_getter version_number : {Int32, Int32} = begin
    version = self.version
    dot = version.index('.') || raise RuntimeError.new("Invalid libpcre2 version") ## THE ERROR!!
    space = version.index(' ', dot) || raise RuntimeError.new("Invalid libpcre2 version")
    # PCRE2 versions can contain -RC{N} which would make `.to_i` fail unless strict is set to false
    {version.byte_slice(0, dot).to_i, version.byte_slice(dot + 1, space - dot - 1).to_i(strict: false)}
  end

You can see that the call is exactly the same as %41.

LibPCRE2.config(LibPCRE2::CONFIG_VERSION, pointer)

Even though size was set to 17, it likely fails because pointer was not properly set, leading to an incorrect string generation. However, what it is really doing is simply getting the pcre2 version.

So, let’s modify /usr/local/share/crystal/src/regex/pcre2.cr as follows:

    version = "10.42 2022-12-11" # self.version

At least this way, pcre2 will no longer be necessary.

crystal build hw1.cr

There are certainly programs that do not need regular expressions. For example, programs that only perform numerical calculations. It might be useful if pcre2 is not linked when it is not needed.

weirdbricks · August 12, 2024, 12:49am

This sounds great to me. Curious: does removing pcre also make the building faster ?

jgaskins · August 12, 2024, 12:55am

Removing dependencies will always build times, but I wouldn’t imagine the time savings would be noticeable in this case. The Crystal PCRE bindings aren’t that big, so they don’t take that long to compile, and the library itself is already compiled, so removing it saves at most a few milliseconds of link time.

kojix2 · August 12, 2024, 1:09am

They are always curious to know what version of libpcre2 we are using. I have sent a pull request to protect our secret from them. Hopefully this issue will be fixed.