RFC: Link configuration

Crystal Contrib
1

Right now the given a @[Link("foo" ... )] statement the compiler will do the best it can to find the library to link.

That process is basically for posix environments that for each foo library:

  1. If pkg-config is installed it will ask to it using pkg-config foo --libs [--static]
  2. Search libfoo.a within /usr/lib, /usr/local/lib

For simple use cases, this is enough. But there are a couple of limitations:

  1. It’s hard for the end user to link to a custom build of the library, for example with debug information.
  2. If a shard requires a library the user has no opinion whether the library can be linked statically or not.
  3. In OSX it is possible to link to a static library in some cases but that requires the full path to libfoo.a file.

We don’t want to lose the simplicity of the build process that involves linking, that’s for sure.

The question is how we can enable customization of the linking and have nice defaults that are populated by the compiler and also probably by the shards.

At the end of the day having some lookups of link configuration files at the: project, shard, user, compiler level seems a good idea to me.

Using the foo at @[Link("foo")] as a key for accessing that configuration seems simple.

Maybe when building a project by the first time a .crystal-link.yml could be generated or expanded with the missing entries. Or a separate step can also provide that expansion directly.

The current arguments to the LinkAnnotation could be used to populate this defaults. As long as there is no other .crystal-link.yml at project, shard, user, compiler that states a value.

The link file could provide directly the flags to use by the linker

version: 1
libs:
  foo: /path/to/libfoo.a
  bar: libbar.a -L/usr/local/lib

Or could eventually support more features like running a bash/script to detect path/version of a specific library.

Or if no library is found it will be able to emit something in the link config file.

A more simple alternative would be to use convention over configuration. An environment variable per library like CRYSTAL_LINK_FOO, if defined, could state the link command to use, if not keep the current behavior. Although it is simple it leaves to the user the burden of managing all those settings.

1 Like
2

I think for a file like this:

@[Link("foo")]
@[Link("maybe something else")]
lib LibFoo
end

the compiler could introduce a new flag, --link that would look like this:

crystal build foo.cr --link "LibFoo=..."

and then the compiler would not use the @[Link] declarations on top of that lib but instead would read from the command line. I think that ... that I put there could have the same syntax as what you put inside @[Link(...)] and the compiler could simply put that inside @[Link(...)], parse it and apply the same logic.

That way you can configure things based on lib names.

Then, on top of that, we can think of a way to write all of those link options somewhere, but probably putting them in a Makefile should be enough for now.

1 Like
3

How do other languages supporting binding declarations handle this?

4

Having a file for this seems super overkill. Traditionally it’s the user’s job to set up the linking environment for the linker (LD_LIBRARY_PATH, PKG_CONFIG_PATH, etc.) and the compiler is agnostic. But I’d accept some easier to use linking overrides through a --link-override flag or CRYSTAL_LINK_OVERRIDES env var, since some configurations make this hard.

2 Likes
5

Right now the compiler does not check for LIBRARY_PATH at all. The only path that will be used are the one informed by pkg-config or /usr/lib, /usr/local/lib.

pkg-config is an optional dependency for crystal. The user can create a custom .pc file to link specific file and even force a static version of the library. But then we end up forcing the user to actually write a “configuration file”.

Using a single environment variable to list directories to look up for lib & static libs based on name should be the simplest for the user. But that still does not cover how we can choose when building whether we prefer static or shared linking. Yet is far more simple than writing a custom .pc file. I think this can be used for libraries embedded by the compiler like pcre, gc, compiler-rt, etc. And letting the wrapper script that calls the compiler define the path as the embedded ones.

In a more flexible scenario I like @asterite proposal of --link to override all @[Link] of a given lib.

But having a configuration file seems more controllable IMO. But it would be good to support the former two aspects (env variable + link option) even if we don’t end up supporting a configuration file.

6

What about adding the libraries to shards.yml and having the compiler search for any linkable libraries inside the ./lib or ./lib/links directory?

7

I think that boils down with the same story as how to customize/override shards declaration per project basis. Which is something I would still like to see.

8

We do use LD_LIBRARY_PATH because when crystal fails to find a library it falls back to -lfoo, and the linker looks at LD_LIBRARY_PATH.

9

Implemented the env var configuration for lookup part at #7562

10

I think I don’t understand the issue, because I never thought linking was a problem:

  • If I want embedded static libraries to be used in a crystal distribution, I specify export LIBRARY_PATH=/path/to/crystal/embedded/lib:$LIBRARY_PATH in the crystal shell wrapper script so the linker will find them —this achieves the same as #7562.

  • If I want to link a specific library installation —for example OpenSSL— if --link-flags isn’t enough or the library provides a pkgconfig file, I tweak the LIBRARY_PATH, LD_LIBRARY_PATH and/or PKG_CONFIG_PATH environment variables; For example:

    $ export LD_LIBRARY_PATH=/opt/openssl-1.1.1/lib:$LD_LIBRARY_PATH
$ export PKG_CONFIG_PATH=/opt/openssl-1.1.1/lib/pkgconfig:$PKG_CONFIG_PATH
$ bin/crystal spec spec/std/openssl

  • If I want to link to a system library in a shard I use @Link("foo") (see stdlib).

  • If I want to link a locally built static library in a shard, I use @Link(ldflags: "#{__DIR__}/ext/libfoo.a") (see scrypt or libcrystal in stdlib);

  • If I wanted to link to a locally build shared library, I’d use @Link(ldflags: "-L#{__DIR__}/ext -lfoo") but using a locally built shared library is a bad idea in a shard, because executing the program requires LD_LIBRARY_PATH to be exported to actually use the library.

11

@ysbaddaden I think the problem we are trying to solve here is that someone wrote @[Link("foo")] in a shard, which you don’t own, but you want to use that shard but link it to something else, maybe using ldflags: ... and a particular object file.

12

Given a @Link("foo", static: true) The current behavior is

  • pkg-config #{libname} --libs --static
  • pkg-config #{libname} --libs
  • look up lib#{libname}.a in /usr/lib, /usr/local/lib

The user can’t change the library linked without changing the source code or writing a .pc and installing/configuring pkg-config.

It is not blocking. But is a way to improve I think the current behavior.

Using LIBRARY_PATH will not override the result of pkg-config so, if pkg-config and the library are installed I think is not enough to:

13

That issue may be related to the topic?

14

Perhaps slightly different but what do you do if the library name doesn’t match the pkg-config name?

An example is libsodium:

lib/libsodium.23.dylib
lib/libsodium.a
lib/libsodium.dylib
lib/libsodium.lalib/pkgconfig/libsodium.pc

# Linking is -lsodium
pkg-config libsodium --modversion
1.0.18

If anyone would like to look further, Travis won’t link with a custom path even with LD_LIBRARY_PATH.
https://github.com/didactic-drunk/cox

I’m posting this here because I’m thoroughly confused about the linking process and how I’m supposed to proceed. This is a one off custom build for testing. I can’t use /usr/local with caching because other files are there which generates a > 450mb cache file.

The type of thing I’m trying to do can’t be that uncommon. Install 1-2 libraries under a custom path and build an application with shards. The process should be simple and painless.

15

@Didactic.Drunk, I guess you want to link to libsodium.a. For linking agains a custom path there are two options.

  1. Prepend lib/libsodium.lalib/pkgconfig/ to PKG_CONFIG_PATH, or
  2. Prepend lib/ to CRYSTAL_LIBRARY_PATH. This var is used in the -L option of the linker.
    2.1. If this fails, try using @[Link("sodium", static: true)] instead of @[Link("sodium")] (while keeping lib/ in CRYSTAL_LIBRARY_PATH)

Ref: https://crystal-lang.org/api/0.29.0/Link.html

16

Got it. Thank you.

17

Is there a way to set environment variables as part of a build? How an I set PKG_CONFIG_PATH or CRYSTAL_LIBRARY_PATH as part of a build or when running crystal spec?

18

VAR=val command should do it.

19

Is there a way to do it automatically as part of the build process or when running crystal spec ; crystal foo ; crystal bar where it doesn’t need to be specified on the command line or before running the command or as a wrapper script?

Is there some way to do it in a macro before pkg-config is run?

{{ setenv  PKG_CONFIG_PATH `./find_package.sh` }}

I’m searching for the most recent package and building a newer one locally as linux packages are often frozen for a particular release. Old packages often won’t work. Forcing the user to manually compile a package and set environment variables before running crystal foo is unworkable.

20

The current option is to either have a Makefile or a shell script that will export the variables you want.

There is no current solution on behalf crystal/shard to install library dependencies in a uniform way, hence neither is a way to find them.

The beginning of this thread was about having a configuration file for the linked options, that instead of been computed directly, could be generated if missing. Allowing the user to override it.