Shipping the interpreter

Crystal Contrib
1

The biggest obstacle to having the interpreter available in distribution package is that it doesn’t work with a compiler statically linked against musl libc. Static musl doesn’t provide libdl which we use for loading dynamic libraries at runtime into the interpreted program.
But we’re using exactly that (statically linked musl) to build our linux distribution binaries.

Enabling interpreter support for our linux packages would require a major change to our distribution packaging system. We would need to recompile (or at least re-link) the compiler binary on every target platform against the specific libraries of that platform (and make sure the necessary libraries are installed).
Avoiding all that is one of the reasons we’re using a statically linked binary. It’s a lot of work to setup and maintain library dependencies for dozens of supported package repositories.

I’d like to avoid that if possible. It’s a very time-consuming responsibility and we could really use the time for actualy language development (instead of dealing with package distribution).

In my opinion, the best solution would be to leave building dynamically linked compiler binaries with out-of-the-box interpreter support to downstream maintainers. This is nothing that the compiler team needs to provide. Packaging can easily be outsourced and ideally taken over by people who actually know how these things work (examples for this are the crystal packages in the Arch Linux and Alpine Linux repositories).
That could be community-supported repositories or official packages sources of distributions and pacakge managers. Either way, it would free up resources from the language maintainers to focus on the primary task of maintaining the language. Maybe we could eventually even fade out our own package sources with statically compiled binaries.
In the ideal case, we would just need to provide a statically linked compiled binary for every target platform. Downstream packages can pick them up and build custom, dynamically linked binaries for their platforms and put them in packages.
That would be the ideal dream world - from a language maintainers perspective. But I’m realistic in knowing that we’re quite far away from that.

Another alterantive would be looking into other options for loading libraries at runtime. Maybe something like this could help: GitHub - mittorn/custom-linker: libdl replacement that allows to load library from static binary
Bonus points if we could also find a way to load static libraries at runtime (libdl can only load dynamic libraries).
But all this really seems to be very much uncharted territory. Apparently, there are very few use cases similar to ours, where we essentially want to link an entire program with all its dependencies, including libc, at runtime and independent of the host program (the compiler itself).

8 Likes
2

I don’t think this is a huge issue. We can maybe ship an interpreter enabled package for a specific popular platform, say only latest Ubuntu and Homebrew or something like that, but pushing for regular distribution package maintainers to package crystal is the way to go. It sounds daunting at first but actually shouldn’t be, we just have to get existing maintainers in the popular distributions get excited about Crystal or a project it’s a dependency of.

Another venue we have is making good user guides on how to build an interpreter enabled compiler from the static linked distribution one. But in the end reframing our statically linked packages as just a basic bootstrap binary might actually be a good thing.

5 Likes
3

I agree with the reframing the static compiler as a bootstrapping artifact. That was one of the initial goals.

Having some articles for building your own crystal package sounds very good! Many times I wanted to address package maintainers. I did that manually. Would be great to have a registry or forum category for specific for this I think.

I think we need to keep shipping a good package for the main distros at least. I thought OBS would simplify that. Custom recipes and dependencies for each distro. If not, maybe the OBS matrix should be shrinked.

My only experience with linux packages is with Crystal and I am biased to think that it’s distributions, at least for a broad number of popular distros is on us.

I didn’t like the idea of saying that something is released, yet the official installation methods might have some considerable delay until they are updated.

I know it’s a matter of scale, and we can’t do it for all. That is hard to draw a line. But I would like to keep some important amount of good packages for main distros.

1 Like
4

In fact, if some where exists document which teach me how to enable interpreter config when build on my local laptop directly use libc?

a REPL like ghci for the Haskell is really help i think.

I try to compile on my arch linux, use following command, it seem broken on link, which cause crystal i just return Error.

 ╰─ $ make crystal interpreter=1 stats=1 
Using /usr/sbin/llvm-config [version= 13.0.1]
g++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/include -std=c++14   -fno-exceptions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="9b8870cfa" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1654088385" CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign --stats  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Parse:                             00:00:00.000071930 (   0.76MB)
Semantic (top level):              00:00:00.874724429 ( 138.93MB)
Semantic (new):                    00:00:00.006505936 ( 138.93MB)
Semantic (type declarations):      00:00:00.103450278 ( 154.93MB)
Semantic (abstract def check):     00:00:00.122829422 ( 154.93MB)
Semantic (ivars initializers):     00:00:20.898831952 (1618.86MB)
Semantic (cvars initializers):     00:00:00.023040574 (1618.86MB)
Semantic (main):                   00:00:23.188529316 (1938.86MB)
Semantic (cleanup):                00:00:00.001307373 (1938.86MB)
Semantic (recursive struct check): 00:00:00.003145153 (1938.86MB)
Codegen (crystal):                 00:00:18.257454312 (2410.86MB)
Codegen (bc+obj):                  00:00:30.971408345 (2410.86MB)
Codegen (linking):                 00:00:08.526845818 (2410.86MB)

Macro runs:
 - /home/common/Git/crystal/src/ecr/process.cr: 00:00:19.007206904

Codegen (bc+obj):
 - no previous .o files were reused
╭─ 22:58  zw963 ⮀ ~/Git/crystal/.build ⮀ ⭠ (0123976d8) master u=  ➦  
 ╰─ $ ./crystal i
Error:
5

You should be running crystal via ./bin/crystal, not ./.build/crystal.

1 Like
6

Yes, it works! thank you very much.

For interest, please see `a Dockerfile + one line sh docker command` for build specified version's crystal static binary only - #7 by zw963

7

Here is a crystal REPL blog by @beta-ziliani , which i even never find it until today, after i built Crystal with interpreter enabled yesterday.

i consider Crystal community is very ignored the existence of REPL of Crystal, Newbile like me, all think REPL is still WIP, and still not usable.

But the in fact is, you can play almost everything simple syntax study use crystal i, and even, we are a debugger add to it, as the debugger added into ruby 3.1 by @ko1.

We are pry like feature in Crystal! even, the promot is pry>, but who(especial newbie) know it?

image
image703×202 13 KB

i am a HEAVY pry user, i even i write gem pryx, just for bundled with some nice feature, and make remote pry can listen use http ip:port when i use ruby to development API within docker container, but, after i read almost all document(except blog), no find clue about if REPL is ready and how to enable it.

So, i consider we must told people, crystal builtin with interpreter, and push people to use it, otherwise, no one use it.

Following is a PR relative to it, but no update since Jan 2022.

1 Like
8

Hello, I guess the interpreter is puts in second plan it’s because it still experimental and is not officially released as supported.

As stated in the “What is the current status?” section of the blog:

The interpreter is currently on an experimental phase, with lots of missing bits. The reason to merge it at this early stage is to enable a proper discussion of interpreter-related PRs and speed up its development a bit. For those willing to try it out, we welcome interpreter-related issues taking into consideration the already known issues aforelinked.

However, If you’re interested I have a super shard wrapping ‘crystal i’, giving a nice interface and some other nice features, like auto-completion, auto-indentation, and actually, basic support for pry!

Though not all is supported, I admit interpreter and pry are often useful for debugging or test small snippet of code ! (thanks a lot @asterite for your work)

I recommend to try my shard as it start to become stable and fully featured :)

1 Like
9

Yes, this is my concern, if not enable it, new guys like me don’t know i can try play code in a REPL, that will help a lot when start to study one new programming language.

Thank you, i will, but have to mention, maybe full featured is not so important, we can do this in next interations, but, when stable enough, we should enable it ASSP from my point.

2 Likes
10

Thanks for making it! I set up a PR that adds a postinstall script, so now it can be added as a legitimate shard and an ic binary will be built and placed in the project’s bin directory.

If the compiler doesn’t currently have the interpreter enabled (and the reasonings made by @straight-shoota at the beginning of this topic make it sound like that won’t be changing soon), then having this shard easily installable is my favorite second-best option.

1 Like
11

Could we at least make it work for Mac? I don’t think we are using libmusl there.

I also don’t understand why what’s the issue with the interpreter and static/dynamic libraries. @straight-shoota could you expand on that?

12

But that’s just the thing: At this point, the interpreter is still in its infancy and has lots of rough edges. Because of that we don’t think it is quite well suited for beginners yet. It’s likely to run into problems caused by the incomplete interpreter implementation and not being able to properly understand them, being a novice user. Thus the interpreter feature is currently intended for advanced users who are already familiar with the language and have some compiler experience.
It’s great if you’re having success with using the interpreter. And thanks for sharing your experiences with it. I’d definitely encourage to play with it, but at this time there are still a couple of extra steps you need to take in order to use the interpreter.

4 Likes
13

Yes, on MacOS distribution of the interpreter should not pose a problem.

We use libdl to load binaries dynamically at runtime. That only works for shared libraries, but not for static libraries or plain object files.

14

Are we using static libraries everywhere? (for the compiler distribution)

15

LLVM bindings use llvm_ext.o. Most of our distribution packages ship a custom build of libgc.a. We have a workaround in place for the latter: crystal/context.cr at 8948404a4debdd25204f81f3e86caee173a950a2 · crystal-lang/crystal · GitHub

1 Like
16

I’m quite late to this thread, but I’m learning more about the interpreter and I’m blown away by how much of a value add it would be to use.

Enabling interpreter support for our linux packages would require a major change to our distribution packaging system. We would need to recompile (or at least re-link) the compiler binary on every target platform against the specific libraries of that platform (and make sure the necessary libraries are installed).
Avoiding all that is one of the reasons we’re using a statically linked binary. It’s a lot of work to setup and maintain library dependencies for dozens of supported package repositories.

Can I help with this? I’m working parallel to this problem now, and I’d really like to make a reproducible way to build, ship (and use) this awesome work. I’m not very knowledgeable on what crystal ships with - the difference between a compiler binary and the binary itself, runtime vs build deps, etc.

Are there docs around this? Looking at All required libraries · crystal-lang/crystal Wiki · GitHub but I’m not sure it captures everything needed for shipping a Crystal interpreter.

Edit: I think I got it. The required libraries are all that I need, but I’m going to build them from source for each platform.

1 Like
17

Following is some simple describe based my own build script (static and dynamic)

static:

at first, you need setup a alpine build environment, check following dockerfile.

dockerfile 
# -*- mode: dockerfile; -*-

# syntax = docker/dockerfile:1.3

# ---
# 1. Use Alpine as base
FROM alpine:3.16 AS base

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories

# ---
# 2. Upgrade system and installed dependencies for security patches
RUN --mount=type=cache,target=/var/cache/apk \
    set -eux; \
    apk upgrade

# ---
# 3. Install common dependencies needed to compile Crystal programs
RUN --mount=type=cache,target=/var/cache/apk \
    set -eux; \
    apk add \
        curl \
        gc-dev \
        gcc \
        git \
        libevent-static \
        musl-dev \
        openssl-dev \
        openssl-libs-static \
        pcre-dev \
        sqlite-static \
        tzdata \
        yaml-static \
        zlib-dev \
        zlib-static \
    ;

# ---
FROM base AS builder

# ---
# 4. Install compiler toolchain and utilities needed to compile Crystal itself
#
# Eg. Alpine's Crystal & Shards
RUN --mount=type=cache,target=/var/cache/apk \
    set -eux; \
    apk add \
        crystal \
        g++ \
        libffi-dev \
        libxml2-dev \
        llvm13-dev \
        llvm13-static \
        make \
        shards \
    ;

then use following command build it.

build static binary 
make_arg='interpreter=1 stats=1 static=1'
        install_target=~/Crystal
        mkdir -p ~/Crystal/sbin
        docker build -t crystal-builder -f $ROOT/../Dockerfile.build_crystal_static --target base .
        docker build -t crystal-builder -f $ROOT/../Dockerfile.build_crystal_static --target builder .
        docker run -it --rm -v $(pwd):/code --workdir /code --user $(id -u):$(id -g) crystal-builder sh -i -c "make clean && make crystal $make_arg && DESTDIR=/code/tmp/crystal_static make install"
        cp tmp/crystal_static/usr/local/bin/crystal $install_target/sbin
        rm -f $install_target/bin/crystal
        cp src/llvm/ext/llvm_ext.o $install_target/share/crystal/src/llvm/ext/llvm_ext.o

dynamic:

build dynamic binary 
install_target=~/Crystal

    pacman install automake \
                 git \
                 libevent \
                 gmp \
                 pcre \
                 openssl \
                 libtool \
                 libxml2 libyaml \
                 llvm lld \
                 wasmer wasmtime \
                 libedit

    mkdir -p output $install_target/bin $install_target/share $install_target/share/crystal/src/llvm/ext/

    make_arg='interpreter=1 stats=1 release=1'
    make clean && make crystal $make_arg && make docs
    rm -rf tmp
    DESTDIR=$PWD/tmp make install

You can check Makefile for details.

3 Likes
18

Thanks!

I think I was overthinking it a bit.
I went ahead and made a tool that aims to produce different kinds of platform packages.

  • dynamically linked with runtime dependencies installed during installation
  • statically linked (not implemented)
  • dynamically linked with all dependencies provided (maybe this would be helpful in some cases? Not tested).

Tool is at: GitHub - skinnyjames/crystal-builder and it’s mostly a proof of concept. I’ve only built debian with it so far, as https://crystal-lang.org/install.sh doesn’t appear to work for Fedora currently.

19

It should be pretty accurate. The interpreter also needs libffi which isn’t mentioned there.

However, this is all a bit besides the point I think. The problem isn’t about compiler dependencies in general. That we got covered and is not really an issue. Maybe it will become a problem when we need to adapt the build process, but then that’s a secondary problem.

The main problem is with a single library, libdl. The interpreter needs that in order to load dynamic libraries at runtime. Usually, this library is included in libc. That’s also the case with musl libc, however it’s unavailable when linking musl statically (which we do for the compiler).

1 Like
20

Do we need libdl? Would it be possible to use something smaller like libltdl?