Shipping the interpreter

The biggest obstacle to having the interpreter available in distribution package is that it doesn’t work with a compiler statically linked against musl libc. Static musl doesn’t provide libdl which we use for loading dynamic libraries at runtime into the interpreted program.
But we’re using exactly that (statically linked musl) to build our linux distribution binaries.

Enabling interpreter support for our linux packages would require a major change to our distribution packaging system. We would need to recompile (or at least re-link) the compiler binary on every target platform against the specific libraries of that platform (and make sure the necessary libraries are installed).
Avoiding all that is one of the reasons we’re using a statically linked binary. It’s a lot of work to setup and maintain library dependencies for dozens of supported package repositories.

I’d like to avoid that if possible. It’s a very time-consuming responsibility and we could really use the time for actualy language development (instead of dealing with package distribution).

In my opinion, the best solution would be to leave building dynamically linked compiler binaries with out-of-the-box interpreter support to downstream maintainers. This is nothing that the compiler team needs to provide. Packaging can easily be outsourced and ideally taken over by people who actually know how these things work (examples for this are the crystal packages in the Arch Linux and Alpine Linux repositories).
That could be community-supported repositories or official packages sources of distributions and pacakge managers. Either way, it would free up resources from the language maintainers to focus on the primary task of maintaining the language. Maybe we could eventually even fade out our own package sources with statically compiled binaries.
In the ideal case, we would just need to provide a statically linked compiled binary for every target platform. Downstream packages can pick them up and build custom, dynamically linked binaries for their platforms and put them in packages.
That would be the ideal dream world - from a language maintainers perspective. But I’m realistic in knowing that we’re quite far away from that.

Another alterantive would be looking into other options for loading libraries at runtime. Maybe something like this could help: GitHub - mittorn/custom-linker: libdl replacement that allows to load library from static binary
Bonus points if we could also find a way to load static libraries at runtime (libdl can only load dynamic libraries).
But all this really seems to be very much uncharted territory. Apparently, there are very few use cases similar to ours, where we essentially want to link an entire program with all its dependencies, including libc, at runtime and independent of the host program (the compiler itself).

5 Likes

I don’t think this is a huge issue. We can maybe ship an interpreter enabled package for a specific popular platform, say only latest Ubuntu and Homebrew or something like that, but pushing for regular distribution package maintainers to package crystal is the way to go. It sounds daunting at first but actually shouldn’t be, we just have to get existing maintainers in the popular distributions get excited about Crystal or a project it’s a dependency of.

Another venue we have is making good user guides on how to build an interpreter enabled compiler from the static linked distribution one. But in the end reframing our statically linked packages as just a basic bootstrap binary might actually be a good thing.

4 Likes

I agree with the reframing the static compiler as a bootstrapping artifact. That was one of the initial goals.

Having some articles for building your own crystal package sounds very good! Many times I wanted to address package maintainers. I did that manually. Would be great to have a registry or forum category for specific for this I think.

I think we need to keep shipping a good package for the main distros at least. I thought OBS would simplify that. Custom recipes and dependencies for each distro. If not, maybe the OBS matrix should be shrinked.

My only experience with linux packages is with Crystal and I am biased to think that it’s distributions, at least for a broad number of popular distros is on us.

I didn’t like the idea of saying that something is released, yet the official installation methods might have some considerable delay until they are updated.

I know it’s a matter of scale, and we can’t do it for all. That is hard to draw a line. But I would like to keep some important amount of good packages for main distros.

1 Like