Hi guys, I have a question (for Linux).
How can I build a crystal library I wrote as a library (.so), and then load it from another crystal program ?
Because I start seriously to think to split my project in libraries, because I am starting to write too an init system (in Crystal as well)
To the best of my knowledge, the best documentation on this topic is here.
Please read Asterite’s comments and the replies to them.
crystal build sum.cr --single-module --link-flags="-shared" -o libsum.so
fun crystal_init : Void
GC.init
LibCrystalMain.__crystal_main(0, Pointer(Pointer(UInt8)).null)
end
fun sum(a : Int32, b : Int32): Int32
puts "Calculate sum"
a + b
end
This only works when there is only one shared library; with two or more, there will be a segmentation fault.
One approach would be to use a single libgc across all libraries while thoroughly mangling other symbols, but this is highly complex. Another idea—related to Ruby’s “external type definitions”—involves generating a type and ID translation table at compile time, though Crystal’s open classes and abstract types complicate things. Finally, there is a possibility that a dedicated shared library format for Crystal, rather than relying on the C ABI, could be devised (though exporting to WebAssembly might become more prevalent).
Since this issue is often discussed, I have written what I have heard so far and what can be inferred from it. Comments from those who know a little more about the issue are expected.
This is currently impossible.
It’s technically possible to create a library from crystal code, but every compilation of a Crystal program will generate the same symbols that will conflict with each other.
We don’t have the means to create a public external API while hiding the internal private symbols.
I think it’s really sad for crystal :X
**Crystal was never meant to compile individual modules, and will likely never be. This is by design.**
Now, my previous comment is kinda outdated today. I then investigated and fixed a couple issues:
But it’s still not possible to compile multiple crystal libraries and link these together unless you give up the stdlib.
Since the OP context is about one Crystal library to be used on another Crystal program, why not just use them as simple shards (in case just separated files in the same project doesn’t solve the problem)?
Why we would need to first compile the Crystal library, since the second program is also in Crystal?
I have now started to use a pattern where I create shards in the same repo. So repo/src is the main code and repo/thing is a shard. I then just use shard.yml to coordinate the dependency.
If in the future the shard is generally useful I can spin it off to its own repo
By “compile individual modules”, are you meaning “build shared libraries” like .so or .dll files?
I was thinking of coming back to Crystal to make an LV2 plugin to do some music stuff in the future should Crystal ever get .so support, but I guess that’s off the table now :-/ It’s unfortunate it likely never will support creating (if I’m understanding this message correctly) shared libs by design unless I give up the stdlib since so much audio stuff is built around shared libs. VSTs, LV2 plugins, heck Audacious plugins. Crystal’s speed and design is nicely suited towards audio, I found out.
Wasn’t there already something like an alternate stdlib without a GC, though? Would that technically allow you to make shared libs, as long as you remember you have no GC?
Well, you can always try to build a library, with an explicit main alternative to initialize the crystal part. As long as the final executable is compatible with all the external libraries that would be brought in, that should work. The GC is probably the main issue because it’s an unique global instance for the whole executable.
We’re fixing issues that prevent building libraries (e.g. internal symbol visibility when `–single-module` is set). We’re removing our dependency on external libraries (e.g. libevent), which can help. We also eventually should have our own GC. All these could help.
As an alternative, there’’s nanolib that I plan to use to write a GC, but it’s a very limited alternative std, without even support for exceptions…
The OP describes using shared Crystal libraries in other Crystal programs, but if it is something like an Android application or a VST plugin where the shared library is the sole thing written in Crystal, then the GC being global wouldn’t be an issue.
The other part to this is the actual loading of .so / .dylib / .dll objects, though. Currently we have Crystal::Loader, but it is an internal component tailor-made for the interpreter. We should publish its API.
So basically you are saying that in the future will be something possible?
At this point, it looks like some use cases are feasible or might be in the future. But no promises.
Maybe someone would like to experiment a bit with this to figure out what works and what could be improvable.
have you looked at what ruby has done New for Ruby 3.4: Modular Garbage Collection and MMTk | Rails at Scale Maybe look at mmtk
This isn’t about using shared Crystal libraries in other Crystal programs, but as MistressRemilia pointed out, there’s some real interest in using them as plugins.
In bioinformatics, bcftools is a well-known command-line tool for working with VCF and BCF files, which store genetic variant information.
bcftools supports user-created plugins implemented as shared libraries. However, its design is quite simple:
For that reason, writing a bcftools plugin in Crystal should work just fine. I haven’t tried it myself yet, but in principle it should be possible.
There might even be other cases like this where Crystal shared libraries can already be used effectively.