Embeddable / Interoperable with ruby

Crystal Contrib
41

Into one library (to be then used with something else). So no matter what it is, to pack everything you want to use into only one library file, by calling crystal build … only once. So instead of having one library with shard A, one library with shard B, one with some own crystal code, … (which wouldn’t work for the reasons you explained) to compile one library file which somewhat bundles shard A, shard B, the own code, etc together. And this one library you use then in your other application.

That’s of course not an option for all use cases. But it’s an option for many of them. And for those it’s really a great thing.

42

Ah, you might be right there. I think it depends on how functions in a module are compiled. If we are generating them in a “private” way so that their mangled name isn’t exposed in the .o file, then each .o file will have its own copy of the function and everything will work fine.

This is my concern. Some fun definitions will exist in both shared libraries. If these functions rely on type IDs, it’s likely things won’t work well.

43

Good point

44

I like the idea of making definition private for each .o! That’s effective to solve the problem of type IDs. But does it mean that each .o file will have their own instance of GC running, or it should be somehow shared anyway?

45

I reply to the most original of this thread.

What you thought almost same as rutie, i guess, the same purpose for use ruby in rust, use rust in ruby.

but, I think the same thing is worth nothing to doing for Crystal, following is reason:

Switch from Ruby to Crystal is quite smoothly if understand ruby well and have some experience with static languages language, so, instead emebeddable/interoperable with ruby, use Crystal directly is a more reasonable option, tough, indeed the ecology around Crystal is far from enough, but, this really the opportunity to port somethings from ruby to crystal, right?

Once you know enough about Crystal, probably you really don’t see the need to complicate this, anyway, instead C extensions, i think the more easy way to rewrite some code(port gem to shard), i really thought, most so-called rewrites are actually copy/paste code from ruby.

One more thing, write two languages source code so similar in one project, mess up easily, ruby use external extension is complicated, maybe use Crystal from Rust is better than ruby.

46

I would assume that there’s no harm in having multiple GC instances, each dedicated to their own space, but we don’t know until we try :smile:

3 Likes
47

About original subject:
I managed to make it work on MacOS. I tried to build and run it in linux, but failed.
The thing I did - I just used Anyolite but in different way. I created a fork that does not initialize ruby, but expects it is already working, and is linked in a different way. GitHub - ahrushetskyi/anyolite: Embedded mruby/Ruby for Crystal

I created a gem with c extension (GitHub - ahrushetskyi/polycrystal: Integrate Crystal into ruby) and put all Anyolite glue there. Had to hack a extconf.rb to compile glue files to *.o, but not link them to extension itself. I was not able to compile glue files outside of extension.

Then, crystal is compiled into .bundle, and loaded by C code from extension. No problems on MacOS at all.
Usage:

class CrystalModule
    class SomeThing
        def some_method
            69.0
        end
    end

    class CrystalClass
        def crystal_method
            return 42
        end

        def return_object
            SomeThing.new
        end

        def recv_arg(arg : Int64) : Void
            puts "Received #{arg}"
        end
    end
end

require 'polycrystal'

Polycrystal::Registry.register(
    # directory with crystal files, added to crystal CRYSTAL_PATH
    path: File.expand_path("#{__dir__}/crystal"),
    # file, that should be loaded. Transformed into `require "sample"` in crystal entrypoint, should be accessible from :path
    file: 'sample.cr', 
    # This gets wrapped by Anyolite
    modules: ['CrystalModule'] 
)

# directory for entrypoint and compiled library. 
build_path = File.expand_path("#{__dir__}/build") 
FileUtils.mkdir_p(build_path)

# compile and load 
# should be called once, after all Polycrystal::Registry#register calls
Polycrystal::Loader.new(build_path: build_path).load

# then, following works
CrystalModule::CrystalClass.new.crystal_method
CrystalModule::CrystalClass.new.return_object
CrystalModule::CrystalClass.new.return_object.some_method
CrystalModule::CrystalClass.new.recv_arg(arg: 42)

Then, I tried to build for linux…

3 Likes
48

That’s actually cool!

It would be awesome to see that functionality for all common platforms.
I might play around with it soon, maybe there’s a way to make it work on Linux (and eventually Windows, too).

Also, feel free to open a pull request to Anyolite, so I can incorporate the option to link to existing Ruby libraries.

49

Build for linux

C extension and glue files compiled without any issues, but I was not able to compile crystal code to dynamic library.

Samle file in question sample.cr:

class CrystalModule
    class SomeThing
        def some_method
            69.0
        end
    end

    class CrystalClass
        def crystal_method
            return 42
        end

        def return_object
            SomeThing.new
        end

        def recv_arg(arg : Int64) : Void
            puts "Received #{arg}"
        end
    end
end

puts CrystalModule::CrystalClass.new.return_object.some_method

crystal build sample.cr --link-flags -shared produces

/usr/bin/ld: /home/crystal/sample: version node not found for symbol *Slice(UInt8)@Slice(T)#initialize:read_only<Pointer(UInt8), (Int32 | UInt32 | UInt64), Bool>:Int32
/usr/bin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status

The same for code with Anyolite, just the name of the symbol in error message is different

crystal build sample.cr -o sample --cross-compile --target "x86_64-unknown-linux-gnu"
# next command is the output from the first command + `-shared`
cc sample.o -o sample.so  -rdynamic -L/usr/bin/../lib/crystal -lpcre -lm -lgc -lpthread -levent -lrt -lpthread -ldl -shared

It produces some *.so file, but I’m not sure it is valid. Because, when I try to do the same with Anyolite code it fails
Entrypoint file (polycrystal_module.cr):

require "anyolite/anyolite"
require "sample"

FAKE_ARG = "polycrystal"

fun __polycrystal_init
    GC.init
    ptr = FAKE_ARG.to_unsafe
    LibCrystalMain.__crystal_main(1, pointerof(ptr))
    puts "Polycrystal init"
end

fun __polycrystal_module_run
    puts "Polycrystal module run"
    rbi = Anyolite::RbInterpreter.new 
    Anyolite::HelperClasses.load_all(rbi)
    Anyolite.wrap rbi, CrystalModule
end

# compile
CRYSTAL_PATH=lib:/usr/bin/../share/crystal/src:/home/sample_project/build/deps:/home/sample_project/crystal ANYOLITE_LINK_GLUE="" crystal build /home/sample_project/build/polycrystal_module.cr -o /home/sample_project/build/polycrystal_module --release -Danyolite_implementation_ruby_3 -Duse_general_object_format_chars -Dexternal_ruby  --cross-compile --target "x86_64-unknown-linux-gnu"
# link
cc /home/sample_project/build/polycrystal_module.o /home/ext/polycrystal/data_helper.o /home/ext/polycrystal/error_helper.o /home/ext/polycrystal/return_functions.o /home/ext/polycrystal/script_helper.o -o /home/sample_project/build/polycrystal_module  -rdynamic -L/usr/bin/../lib/crystal  -lpcre -lm -lgc -lpthread -levent -lrt -lpthread -ldl -lruby

Produces valid executable, it runs. But with -shared it fails during linking with

/usr/bin/ld: /home/sample_project/build/polycrystal_module: version node not found for symbol ~proc9Proc(Int64, Pointer(Anyolite::RbCore::RbValue), Anyolite::RbCore::RbValue, Anyolite::RbCore::RbValue)@build/polycrystal_module.cr:17
/usr/bin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status

I tried a lot of things, but I have no clues.
I thought, it requires ruby to be compiled with clang, but it shows the same error even when -lruby is removed.
Google is not helpfull either. node(js), version, symbol, link is too frequently used elsewhere

Last random attempt, and seems like it works with
--link-flag "-shared -Wl,--version-script=#{mapfile}", where mapfile is

VERS_1.1 {
	global:
		*;
};
1 Like
50

Probably doesn’t solve it, but I strongly recommend to try to build just a very oversimplified ruby extension, to make sure the issue is even related to your Crystal experiments.

Like I was fighting with my weird crashes, which I had no explanation for, as my crystal library would work perfectly fine when I would use it in a C application (turned out I used accidentally all the time the wrong libruby of a different ruby version). In your case it might be some incompatibility between libraries your ruby uses, and your crystal library uses, or something like that - that’s been often a challenge on linux for me.

For my part: to my surprise my stuff work has been working on linux without issues or modifications needed (well, obviously I have to link to the libraries and not just to the framework, and I can’t build a .bundle but have to build a .so), except that I can’t build a crystal-library with all its parts included (so I need to have itself still being dynamically linked to the other libraries like -lz, -lpcre, …).

51

Third party interop won’t cut it actually. We need something like TypeScript. If Crystal could be a layer like type checked language that targets Ruby, things would have been much better. It can optimize code based on types and generate much more efficient ruby code. Native interop is anyway required for this language to grow on people.

Let’s face it. If we want to start a new project today and want a type checked language, we have so many options with great ecosystem like Golang, TypeScript, Java, .NET Core just to name a few default choices. Even Python has great type support during static analysis. This project is only very attractive to existing Rails developers who are growing existing Ruby codebases. That’s the best use case to solve.

Second, let’s learn from Ruby’s growth. The language grew because there was nothing like Rails in other languages at the time, soon this language picked up. So really, it can focus on a new kind of application or introduce a new programming model for which people don’t mind using this language instead of their favorite one.

Great work overall. Love the effort. Want to see this language do well and succeed

52

At mine we are speeding up some Ruby parts using Crystal, but invoking a crystal program and passing few records ids from some table it needs. Even the crystal code needing to boot up, connect to a redis server, connect to a postgres database and fetch things with no cache at all it still way faster than the previous pure ruby implementation. i.e. it’s fast enough for us and with much room for optimizations when needed.

8 Likes
53

I want to use Crystal language with Ruby, and the info in this awesome thread is really helpful. But it’s hard to understand. Luckily, ChatGPT 4 helped me get the main points of the talk between Asterite and Beta-ziliani. Here’s a simple summary of their discussion from ChatGPT.

Asterite explained that creating shared libraries in Crystal is difficult because the type IDs generated in the code are not consistent across different compilations. This inconsistency can lead to issues such as segmentation faults.

Beta-ziliani suggested that for well-defined interfaces of primitive values, similar to the restrictions of the C ABI, the issues might not occur. In response, Asterite acknowledged that functions using primitive types would work well, but if those functions call other non-primitive type functions that rely on type IDs, issues might arise when used in other Crystal shared libraries.

Beta-ziliani then proposed that if each part is compiled as a single module, each object file (.o file) would be self-consistent, including all the required stuff from the standard library. Asterite agreed that if functions within a module are generated privately so their mangled names aren’t exposed in the .o file, each .o file would have its own copy of the function, and everything would work fine.

OK, I understand there will be some limitations. I would like to use Ruby-FFI to call a Crystal shared library, which would have an interface with similar constraints to the C ABI for primitive values. (We might still be able to exchange objects between Ruby and Crystal to some extent by using JSON-serialized strings.) This would enable the use of a single Crystal extension. However, we also need to consider the possibility of using multiple Crystal extensions, especially as creating Ruby extensions in Crystal may become more common in the future. To avoid function conflicts, we might need to make functions private. How can we accomplish this? Is it just a theoretical concept, or are there compiler options that can make it possible? Or do I need to use a more complicated method, like outputting an LLVM IR and compiling it myself?

54

It would be a pity to completely give up the idea of calling Crystal functions from Ruby.

My vague understanding is that the discussion here is that using multiple shared libraries will cause conflicts and problems because the assigned type IDs are different for each compilation.

So what if there is only one compilation?
All necessary crystal libraries are collected and compiled to create only one shared library.

Will this still cause problems? :thinking:

(Of course, you cannot include the Crystal code in Gem. Multiple Crystal gems will be compiled independently, which will cause type ID conflicts. )

55

One simple and performant strategy is to create a socket server on the Crystal program and use FlatBuffers (repo and benchmark), from Google, instead of JSON or protobuf.

This approach is at least 3700x faster than protobuf and more than 7000x faster than json.

2 Likes
56

If someone creates a Crystal binding library to FlatBuffers (available in C and C++), pleae let us know.

1 Like