Embeddable / Interoperable with ruby

Crystal is lacking in popularity and the adoption rate is still low. I’m a ruby developer and we need features provided by crystal in our, now larger, codebase, but it’s hard to justify rewriting to Crystal all the business logic that already works. Even extracting parts, that will benefit the most, is hard to justify because Crystal infrastructure and libraries are way less mature than ruby.
That’s why I think that ability to embed Crystal code in ruby projects will significantly benefit Crystal popularity and adoption rate. It will do the same function that Sorbet Compiler: An experimental, ahead-of-time compiler for Ruby · Sorbet do but in a much better and cleaner way

IMHO: it should be just a gem that contains all required glue logic in ruby and Crystal. It should invoke the compiler (or even have it embedded) with appropriate parameters to compile the code to the ruby extension and load the extension. It should just search for *.cr files inside a project and compile them on boot or on require from ruby code
AOT option should also be available.

It should not be fully smooth or transparent, it should be easy to use and understand. It should not try to provide guarantees that are hard to implement and maintain. There is not much need to make Ruby->Crystal and Crystal->Ruby type conversions 100% smooth and compatible. e.g. when a ruby string is passed to Crystal function, it may be just copied to Crystal string and the original can’t be modified in Crystal.
Ruby classes can be represented in Crystal code as generic RbObject class and use method_missing macro to invoke ruby methods. RBS definitions may be used to generate more specific wrappers, but I doubt that it is worth the effort to implement and maintain.
Conversions for the most popular types from the Ruby standard library should be implemented.

# in ruby:
class SomeRubyClass
  def some_method(param)
    param + 42
  end
end
CrystalClass.new.rb_method(SomeRubyClass.new)
# in crystal
@[ExposeClass]
class CrystalClass
  def rb_method(in : RbObject)
    in.some_method(23)
  end
end

It’ll be hard to expose all Crystal classes and methods to ruby. As far as I understand, JIT compilation will be required to generate code for methods with unknown parameter types.
For simplicity, Crystal classes, that should be exposed to ruby code, should be annotated, and exposed methods should either have all argument types defined or be annotated with possible argument types. As all exposed methods may only receive and return exposed types, most popular types from Crystal standard library should be exposed or conversions to ruby types should be implemented.

@[ExposeClass]
class SampleClass
  # exposed, no params
  def do_smth
    1
  end

  # exposed, param type is defined
  def add42(num : Int32)
    num+42
  end

  # exposed, param type is defined in the annotation
  @[ExposeMethod(num : Float)]
  def add69(num)
    num+69
  end
end
# in ruby
require 'sample_class'
s = SampleClass.new
s.add42(1)

Other things to be done:

Compile to ruby extension. It should be possible to load one or multiple extensions compiled from crystal code. As far as I understand, distributed gem should load Crystal context (code, that is now executed before Crystal main) on boot and load the extension later during boot or on demand
GC interoperability - No idea how to achieve that. Maybe, both GCs may be used simultaneously and objects may just be marked “static”, while they are owned by wrapper objects of the other language. Or maybe crystal can use ruby GC
Make Threads, and Fibers interoperable, handle ruby GIL

3 Likes

There is this project: Anyolite 1.0.0

2 Likes

Anyolite is kinda backward to what is required. It embeds ruby into Crystal, and that is mruby by default.
It fails two requirements, which is a deal breaker for adoption. One - It is not easy to use, it runs mruby by default and has no complete MRI support. Two: most ruby folks run rails. If it can’t run rails, it’s useless.

That’s why we need the opposite. We need to embed crystal into ruby, to ensure that existing ruby code works and ruby infrastructure can be used.

2 Likes

This sounds like a noble goal but I don’t see this something the core team would work on. Crystal and Ruby are different languages and a goal of Crystal itself is not being compatible with Ruby. If you or someone else wants to work on a third party compatibility layer or something that could be handy, but I wouldn’t bank on it being an official project. Just my 2 cents.

3 Likes

Anyolite dev here.

It should technically be possible to run gems like Rails together with Anyolite, but I just never tested it (some example gems run without problems, though). The MRI implementation might be experimental, but the functionality should be completely there.
The major disadvantage here is that you’d need to run some extra setup work - but in the end you should be perfectly able to call Crystal functions from a regular Ruby interpreter using Anyolite.

Talking about disadvantages, you wrote that Anyolite is not easy to use, do you want to elaborate on that?
I’m always glad to hear some criticism, so I can improve Anyolite.

5 Likes

Would it work to simply call a Crystal executable from Ruby? There are ways to speed up the I/O, if that would be relevant to your use case. I think that calling a Crystal executable or communicating with a long-running Crystal server is probably the most viable way to interoperate between a program running in another language and Crystal code.

All that said, I’m not a ruby dev nor a web dev, so maybe there are some requirements of your use case that I’m not considering.

1 Like

It should be possible, but it sounds extremely inefficient, since you’d need a way to convert data from Ruby to Crystal and vice versa. Using pure Ruby might actually be faster at that point.

If there’d be a way to export Crystal functions to other programming languages, this would actually be interesting, though, but things like the Garbage Collector are making this very difficult.

1 Like

At my workplace we use Haskell to speed up some Ruby parts. It’s all web. The way we did it is by having a Haskell server and invoking those endpoints from Ruby. It’s still faster than Ruby. Data is serialized over JSON, though I bet using Protobuf or something else might even be more efficient.

I’d go with that approach for now. The nice thing about that is that if for some reason Crystal (or whatever language you choose) doesn’t work out for you, you can switch to another language very easily: just HTTP and JSON.

There are big issues with how Crystal represents types at runtime that makes it impossible to expose functions from Crystal to other languages. Later in the week I’ll write a post about this. Nothing that can’t be solved, but the current situation prohibits that.

8 Likes

(Another note: we use Haskell because the company members like Haskell. If it were for me I’d immediately switch over to Crystal. I don’t like Haskell that much…)

6 Likes

If there’d be a way to export Crystal functions to other programming languages

It mean compile crystal function to webassembly and the call webassembly function in Ruby?

I am mostly in the same boat like you and am following this thread with much interest.

I’ve been able to use Crystal functions within (Objective) C/C++ code without any issues (so far at least). But for whatever reason I’m not able to use them within ruby (not via ffi nor via extensions). Whenever I try to do so ruby crashes (right when I would set up the crystal environment), but I don’t think ruby is to blame for it. I would clearly think that it’s my fault, on the other hand, those libraries work just fine within C/C++/etc, and there is a chance I’m just doing the ruby part incorrectly.

I’ve been able to use Crystal functions within (Objective) C/C++ code without any issues (so far at least).

May I ask how?
I wouldn’t even know where to start.

But be warned: everyone who is more experienced and skilled might read my following instructions in disgust or horror. So I wouldn’t recommend any of it. But what can I say: for me it has been working so far, without any issues and stable. Something done properly is for sure best, but if “properly” isn’t available then in many cases “it works” is still a great option.

You have a few options how you want to handle the applications start. It’s the easiest if your application starts within crystal-code (this sounds less flexible than it actually is: your other code (for the rest of this post let’s assume it’s C, as it’s easier to type “C” instead of “other language”), so your crystal code, could just call C’s main function, which you simply call notMain and the only thing you do at the start in crystal is to call C’s notMain (you do that just via the lib… fun … stuff as crystal offers it). For the application’s user it’s the same like if the application started with the C code, but it’s a lot easier this way.

Then the crystal methods, which you want to call from within C you have to define not via def but with fun in crystal. In C, you only write the definition for those methods, but no implementation. When you want to link the code, the linker might complain about missing symbols. If that’s the case you have to fix the name mangling (I’m pretty sure this could be automated via some macros, but it’s so easy, that writing a macro is probably not worth the effort). Just look via “nm” what name your C object needs, and add it as an external name in your crystal code (like fun myMethodsNormalName=“myMethodsMangledName”). Nice side effect: once you are able to get the mangled names right by yourself, you don’t even need to create an object file for your C code, but can just let the crystal build command take your C files as linker flags (@[Link(ldflag:“mysource.c”)] and it get’s all compiled just within your usual build command (at least if you use just clang as your linker anyway - as those files would be just compiled and linked in the same step, when crystal just intended to link libraries). If it was crystal which started your application (as in it was crystal’s main method which got started in the first place) you shouldn’t need to do anything else.

Kinda think of it not as using libraries. Just as if it would be all one language. And you declare and implement things just in separate files (especially if you view it not like with Crystal where those two steps are just one, but like you have it with C usually anyway with header files). You declare things on both sides, so you can use them in both worlds, but you implement it only in one (as often as possible in Crystal as it’s easy and we love it, and as often as necessary in C or whatever).

You also can start your application within C, but you will have to start Crystal’s GC.init and LibCrystalMain.__crystal_main in the beginning (=before you call anything else of your crystal code) or you will get segmentation faults as soon as you do anything relevant. Also crystals main method might upset your linker (as in having suddenly two main symbols). But simply rename again your C’s main to notMain, and tell the linker, to use that as the starting method (-e "_notMain").

If you work with classes, you will get quickly frustrated with all the helper methods you have to introduce. You absolutely can implement C class methods in crystal (top level) methods directly, just like top level methods once you fixed the mangled names. If you want to call instance methods, or do Crystal class methods, it is still possible, but requires to tinker with crystal’s compiler (to do this it’s enough to comment out one line - you will create double the amounts of symbols, but the additional symbols won’t conflict with any relevant symbols, so they won’t bother you much).
To write the instance methods of one language in another, it needs a lot more tinkering (it works though; I am currently trying to figure out, how this could be done properly (without generating dozens of unnecessary symbols) and “magically” doing the addressing of the right instance automatically - it works, but at least I need to do a lot of manual stuff to get it working (someone who knows what they are doing, might handle this probably a lot easier - I’ve no experience at all in any of this)).
If the instance would have any variables, you will have to decide for each class on one language where you want to have them “living”, allocated and eventually released (it can be either, but if you mix it, it get’s very quickly messy, and it will be already messy enough, so just pick one side for the better). You can still access the variables in either language (just not directly, but only via getter/setter methods, which doesn’t make a difference anyway other then you loose the public/private/… choice; those getters/setters you just access again like any other instance method).

If you plan on simply wrapping your crystal code in a C library to call it from ruby - I’ve tried exactly this and sadly it didn’t work (but again - it’s not impossible, that I’m doing something wrong on ruby’s side, as I’m really an absolutely noob when it comes to libraries and linking and wrapping). Within the C-ish family though it seems to work pretty well, especially if you stay top-level on both sides (C class methods are also okay). Instance methods, variables, or even just Crystal class methods, kinda work, but for me so far only with modification of the compiler (for now, you probably better just use auxiliary methods to get things to top level).

Oh, and you should have all your Crystal things within one crystal application (can be dozens of files, but I mean, don’t use it like you would usually do with modules; you will use only one crystal module, but it can contain hundreds of unrelated functions) which was built just with one crystal build command.

It’s already past midnight on my side. Tomorrow I’ll upload some small example to play with.

4 Likes

You are right, I was horrified :stuck_out_tongue: This is an incredibly wild thing to undertake but I’m also very impressed that someone tried this and got something to work in some form.

I look forward w/ perverse interest in whatever example you have to share :grin:

4 Likes

No need to be impressed - I’m really not doing much. If one starts the whole thing still via crystal code, then it’s actually not even something which isn’t offered by crystal anyway and could only be seen as syntactic sugar (if even).

As I said yesterday I would explain it all with some code, here it comes.

So the following works fine for me (on osx 10.14 and some recent alpine linux):
Let’s start on the C side with some very standard code:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[]){

  return 0;
}

And let’s say we would really like to use two functions from Crystal, because they are really tricky to implement, so we would prefer to make life easier and more enjoyable by writing them in Crystal. Let’s call those functions foo and throwDice.

def foo(i)
  puts "foo!"
  (1..i).each do |j|
    puts "foo #{j}"
  end
  puts "foo!!!"
end

def throwDice 
  1_u8 + Random::Secure.rand(6)
end

(now that you have seen foo you understand why I didn’t want to write that in C :dizzy_face:)

Still absolutely to the standard so far. Same goes for having to tell the C code about them in order to call them. And let’s also directly use them in C:

#include <stdio.h>
#include <stdlib.h>

void foo(int);
char throwDice();

int main(int argc, char* argv[]){
  foo(10);
  printf("Alea iacta est:  %i!\n", throwDice());
  return 0;
}

From C’s point of view, the only thing that’s missing is the implementation of the two functions (which would be normally maybe anyway stored in other files, something like foo.c and gambling.c). So still nothing unusual.

If you would try to compile the code already, the linker would be complaining about foo and throwDicing being referenced but missing.

Our Crystal code has those two functions, so let’s just put things together.
If you would try to link the object files of Crystal with the one of C, among other issues the linker would still complain about those two methods being missing. You first have to make them visible or whatever (I actually haven’t even checked why their symbols aren’t there). Anyway, you have to make some small changes to your Crystal code and that’s where the fun begins (pun intended) change the defs to funs of all your Crystal functions you want to use in C. And while you are at it, set the types the usual lib-fun way, as if it were for the other direction.

fun foo(i : LibC::Int) : Void
  puts "foo!"
  (1..i).each do |j|
    puts "foo #{j}"
  end
  puts "foo!!!"
end

fun throwDice() : UInt8
  1_u8 + Random::Secure.rand(6)
end

Still no big act (and I believe we still move within what’s intended by Crystal, but I’m not 100% sure).

If you would now again compile everything to object files and try to link them, it would still not work, but at least the symbols of the two functions would be no longer missing - that part is done. But as I said, there is still some issue, we have to main functions and the linker doesn’t like duplicates. So let’s rename the one of C simply to otherMain. If you would re-compile and try again to link, it would actually work. Or sort of. You receive your executable, but when you start it nothing happens, not even an error, and it just ends immediately (at least successfully tho).
And that’s kinda absolutely what we would have to expect, as there is nothing going on in our Crystal code, it starts, there is nothing to do and it ends. otherMain would actually work but it never gets called. But we wanted it to run the C code after all, or we wouldn’t have bothered about using crystal code in C in the first place.

It’s easy so. Just add to your crystal code that you want to run otherMain. And that’s actually all we want Crystal to run here. As Crystal wouldn’t know of otherMain, just do it like with any other external stuff you would want to run in Crystal. With the Link annotation, we can be cheeky and even give it our source (c.c here) instead of the usual library name (you don’t HAVE to do that. This is really optional. But if you do, you can later just do a crystal build myCrystalFile.cr (as well as --static/--release/etc as you wish) and all will be done at once; if your c code requires more files or any flags, jsut add them within the string of the annotation). And we would really like everything to behave as if it was all C we started, so we also take care of the exit code and ARGV. And our Crystal code then looks like this:

@[Link(ldflags:"#{__DIR__}/c.c")]
lib LibOther
  fun otherMain(LibC::Int, UInt8**) : LibC::Int
end

fun foo(i : LibC::Int) : Void
  puts "foo!"
  (1..i).each do |j|
    puts "foo #{j}"
  end
  puts "foo!!!"
end

fun throwDice() : UInt8
  1_u8 + Random::Secure.rand(6)
end

exit LibOther.otherMain(ARGV.size, ARGV.map(&.to_unsafe))

And that’s it, nothing else left to be done. But feel free to add other exciting Crystal functions if foo and throwDice aren’t exciting enough for you. You should just stay within one Crystal application (as you would normally create one executable by calling the crystal command just once) - this one crystal application can contain dozens of unrelated tasks or shards tho.

Run your crystal build command and enjoy. Your C code looks almost perfectly normal (only difference is the renamed main, that’s it). Your Crystal code has a tiny bit of boilerplate, but is also still very much the way as it would usually look like. I’m happy with this solution and maybe you will be, too.

You could even go and build a static crystal library which you then link with whatever C project you might have (then you basically are really at the point of calling it your very standard library behaviour as one would expect). And on osx there is nothing else you would have to adjust, just as it is, go and do so (choose cross-compiling, and then use the Crystal object file like you would usually do in order to create your static dylib or bundle). On linux I wasn’t able to do so, but that’s probably just because I wouldn’t know how to create a static (working!) library including Crystal’s dependencies (-lz -lpcre etc) so that issue seems not necessarily related to the mixing of languages :man_shrugging:

Oh, and just in case: It doesn’t just work for such awesome things like foo or a random number between 1 and 6. It works just the same with such trivial or boring things like Kemal or HTTP::Client :crazy_face:

1 Like

Now if you want to use Crystal code in combination with C++, you will run into name mangling issues.
myfunction usually becomes _ followed by the length of the function’s name followed by the function’s name (plus usually some stuff for the types if you handle data - but more about this later), so in this case _10myfunction. If it’s in a namespace or a class or something, anything which you would call something like Foo::Bar::Baz etc then it’s like _ZN#{name.split("::").map{|x| "#{x.size}#{x}"}.join}E so in this case it would be _ZN3Foo3Bar3BazE (again likely with some encoded data types quite in the beginning and or at the end). It’s definitely useful to know and understand it so far. But don’t bother with the details. Generate the object file for your C++ code and just look what symbols it expects. It get’s even easier if you temporarily just add a CRYSTAL suffix to the method name and filter with grep (but don’t forget to subtract the digits of CRYSTAL of the length). And then just take the values you find this way. If you have methods with the same names which are only different by the data types, either help yourself with some temporary suffix (as it won’t change the data type identifiers) or even go and get those mangled names demangled by something like (llvm-)cxxfilt. If you are willing to, you could also write your self a precompiler, which first looks all necessary method names up in the C++ object file, translates them with cxxfilt and adds the mangled names to your crystal code (or if you are extremely bored, write a macro which results in the same, but I absolutely don’t think it’s worth it, after a few times, you will guess and add most of the mangled names correctly yourself while you write the code).

Oh, and you use the mangled names in your Crystal code like this

fun myAwesomeMethod="INSERT MANGLED NAME HERE"(…) : …
 …
 …
end

So if you wanted to write a new class method bar for the C++ class Foo, your code would look something like this (the exact mangled name depends on the data types your sharing around):

fun thisNameDoesntMatterIfYouWillCallItOnlyFromCPP="_ZN3Foo3BarE"(…) : …
 …
end

The name mangling thing here is NOT valid if you use MSVC though (in general the whole procedure is the same, just the mangling will follow a different scheme, which you would have to figure out for yourself). But AFAIK (which doesn’t have to mean much) all other compilers follow this pattern (if not then you will find out eventually). But even if you should compile your stuff on Windows, just use clang and everything should be fine :man_shrugging:

I am unsure whether this will actually work without memory leaks and similar problems (it might actually, but I haven’t tested it).

Definitely a creative method, but too much setup work and boilerplate for my taste :slight_smile:

The boilerplate are just 5 lines

@[Link(ldflags:"#{__DIR__}/c.c")]
lib LibOther
  fun otherMain(LibC::Int, UInt8**) : LibC::Int
end
exit LibOther.otherMain(ARGV.size, ARGV.map(&.to_unsafe))

while one basically can’t even count the first line (as it is convenience not to have to call a second command - which one would have to call with a “normal” library as well).
Otherwise it’s just renaming the originally main to otherMain and replacing def with fun on (everything else is exactly the same as it would be if there was something oob to build libraries).

Memory leaks are indeed possible and something one has to take some care, but it’s not more (or less for that matter) as it’s with libraries in general (actually, you likely will have to worry more about memory being freed possibly too early than about leaks, because Crystal’s GC is still running, it just won’t know how long you might use some of the data in C land - but again, this issue exists in general if you handle data outside of its natural scope).

1 Like

My adventures with the subject:
I was able to compile Crystal code to shared library (MacOS dylib), link it with some glue code in C and load it as C extension in ruby. I was able to call Crystal fun from ruby, and that fun was able to instantiate object and call instance method on that object. Looks like GC works. No segfault for now
Crystal file:

class SomeClass
    def do_something(num : Int32)
        num*3+3
    end
end

FAKE_ARG = "crystal"

fun __crystal_init
    GC.init
    ptr = FAKE_ARG.to_unsafe
    LibCrystalMain.__crystal_main(1, pointerof(ptr))
end

fun __crystal_module_run
    puts "Crystal module run"
    test_add(1, 2)
end

def test_add(x, y)
    r = x + y + 3
    res = SomeClass.new().do_something(r)

    puts "Got #{r} and #{res}"
    res
end

res = test_add(1, 2)

puts res

Compile with

crystal build test.cr --link-flags "-shared" -o libcrystal_module.dylib

Ruby c extension code:

#include <ruby.h>
#include "extconf.h"

extern void __crystal_init();
extern void __crystal_module_run();


VALUE test_cfunc(VALUE _self, VALUE num) {
    int n = NUM2INT(num);
    VALUE result = INT2NUM(n+1);
    return result;
}

VALUE test_module_run(VALUE _self) {
    __crystal_module_run();
    return Qnil;
}


void Init_crystal() {
    VALUE mod = rb_define_module("TestModule");
    __crystal_init();
    rb_define_module_function(mod, "test", test_cfunc, 1);
    rb_define_module_function(mod, "dry_run", test_module_run, 0);
}

To link shared library to ruby C extension, I had to add

$LDFLAGS << " -L#{File.expand_path("#{__dir__}/../../crystal",)} -lcrystal_module"

to extconf.rb

Everything works as expected, but that is not much

Next, I need to process Crystal code and generate ruby wrappers, and load code dynamically.
First of all, I’ll try to compile Crystal code to MacOS bundle file and load it using dlload. I’ll add one more fun to Crystal code (lets call it __crystal_load_module) that will add Crystal wrappers to ruby.
I looked how Anyolite does that, and I’m thinking how can I either implement something similar or reuse existing code. I’m impressed what they were able to achieve with crippled Crystal macros and annotations.
I can’t link Crystal code to ruby API dorectly, so I think I’ll pass a struct with ruby API function pointers to __crystal_load_module.

2 Likes

As far as I know (very little experience in cpp),

extern "C" {
  int some_func(int arg){ 
    return arg +1;
  }
}

should produce unmangled symbol names (e.g. some_func or _some_func).
Is there any caveats with extern "C" and Crystal?

2 Likes