Crystal in (possible) WASM everywhere future?

1

I’ve just read this article on a role a WASM could play outside Web browsers which I’ve personally found very interesting for a lot of reasons.

It’s reminded me of this super fun talk which I also highly recommend to anyone (as any other talks/screencasts by Gary Bernhardt).

https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript

It’s like the future described in this talk is coming true step by step :slight_smile:

So I guess I have a lot of thoughts after reading the article right now, but mostly I’m thinking of this:

How Crystal with implicit imports borrowed from Ruby and a global type inference can fit into a model described in this Mozilla article - “nanoprocesses”?

For those of you who don’t want to read the whole article, the “nanoprocess” is a very lightweight way to isolate a piece of code for security reasons and an explicit way to request/provide access to resources like files, sockets and anything that should be protected. So it’s a way to implement the principle of least authority

4 Likes
2

I was thinking and write some comments on Trion OS forum on SourceForge for about similar stuff back in May 2004 . Universal bytecode that can be executed on any device, desktop IoT device and so on. Actually looks like I am father of Docker and Kubernetes ;) At least I coned that idea way back before actual implementation. May be they read my messages on that forum somehow ?)

Here is that message:
https://sourceforge.net/p/trion/mailman/message/2075631/

1 Like
3

Well I’m skeptical at the moment. Universal bytecode is the JVM all over again, let’s hope they have learnt from history. As for those “nanoprocesses” with fine-grained permissions and data copies I don’t think they’ll ever make it work outside their bubble, but I guess we’ll just have to wait and see.

I want WASM to replace js in the browser, that’s as far as my needs go.

1 Like
4

reinvent java

2 Likes
5

Except that JVM have failed to hold “write once, run everywhere” promise, while JavaScript succeeded more on that front. And WASM is the next step.

The thing with this “terrible terrible JavaScript” is that it’s now everywhere and is probably the most used programming language. Turns out it didn’t even needed to be the best programming language for that to happen. But it was first, and then multiple vendors had to agree and implement it and all other features of evolving Web platform.

I see WASM as a very similar thing. It will be implemented by all browser vendors, it will work everywhere (Web). To bring it outside the Web is a logical step forward given the amount of “resources” available to that ecosystem (number of platforms, libraries, developers, you name it).

I would probably worry about Web itself to be on decline (watched one talk, but can’t remember the title from the top of my head), which of course can affect the WASM story outside of the Web.

6

I agree, it’s too early to make any conclusions and as everything in this world it’s hard to predict. But at least WASM itself is showing a great progress and thinking about Crystal’s future in the bigger context is also important.

7

Maybe:) OTOH it’s not that uncommon for different people to come up with very similar ideas at roughly the same time.

8

Idea was described at least 6 years before first Docker release and 10 years before Kubernetes first mentioning ;)

Idea is more low level though as it was for grid/cloud operating system. Can be done with LLVM BC easily. Wasm just one step above. All current browsers are using JIT somehow and most likely they use LLVM for that.

9

Meanwhile WebAssembly 1.0 Becomes a W3C Recommendation and the Fourth Language to Run Natively in Browsers

Support for WebAssembly in browsers is already great.

Other great news is that things needed for Crystal to support WASM target such as

  • Threads
  • Exception handling
  • Garbage collection

are already in progress. See Features to add after the MVP

Fingers crossed for fast progress in this area. Can’t wait to use Crystal in browser or any other WASM runtime.

10

Hello, I am new here i was searching around for language with good support for Web Assembly and i am big ruby fan so are there any plans in near future to target WASM ?

1 Like
11

Have a fun read :D https://github.com/crystal-lang/crystal/issues/829

3 Likes
12

I would love to get WASM running. I did a little bit with it, but nothing really usable. I came across this gist https://github.com/crystal-lang/crystal/issues/829#issuecomment-599596803 that shows. My guess is if you compile crystal with LLVM10, and maybe a few flags, then you might be able to get away with it using the wasm target triple.

With that said, I have no clue what I’m doing, so I could also be talking gibberish lol. If you happen to understand any of this, or manage to get anything going, report back!

2 Likes
13

hello, I am new for llvm and webassembly, but I tried this:

$crystal --version
Crystal 0.36.1 [c3a3c1823] (2021-02-02)

LLVM: 10.0.0
Default target: x86_64-unknown-linux-gnu

$../llvm/llvm-project/build/bin/llc --version
LLVM (http://llvm.org/):
  LLVM version 13.0.0git
  DEBUG build with assertions.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: sandybridge

  Registered Targets:
    wasm32 - WebAssembly 32-bit
    wasm64 - WebAssembly 64-bit

$cat hello.cr
puts "hello world"

$crystal  build hello.cr  --emit llvm-bc  # output hello.bc
$crystal  build hello.cr  --emit llvm-ir  # output hello.ll
$ crystal  build hello.cr  --emit asm # output hello.s

$ ../binaryen/bin/eosio-s2wasm  hello.s -o hello.wast #  eosio-s2wasm is from https://github.com/EOSIO/binaryen

[[function element:]]:
==========
.cfi_startproc
	subq	$184, %rsp
	.cfi_def_cfa_offset 192
	mo
==========
Aborted (core dumped)




$../llvm/llvm-project/build/bin/llc  hello.ll -march=wasm32
<inline asm>:2:13: error: Unexpected token in operand: %
      pushq %rdi        // push 1st argument (because of initial resume)
            ^
<inline asm>:3:13: error: Unexpected token in operand: %
      pushq %rbx        // push callee-saved registers on the stack
            ^
<inline asm>:4:13: error: Unexpected token in operand: %
      pushq %rbp
            ^
<inline asm>:5:13: error: Unexpected token in operand: %
      pushq %r12
            ^
<inline asm>:6:13: error: Unexpected token in operand: %
      pushq %r13
            ^
<inline asm>:7:13: error: Unexpected token in operand: %
      pushq %r14
            ^
<inline asm>:8:13: error: Unexpected token in operand: %
      pushq %r15
            ^
<inline asm>:9:12: error: Unexpected token in operand: %
      movq %rsp, 0(%rdi)  // current_context.stack_top = %rsp
           ^
<inline asm>:10:12: error: Unexpected token in operand: $
      movl $1, 8(%rdi)   // current_context.resumable = 1
           ^
<inline asm>:12:12: error: Unexpected token in operand: $
      movl $0, 8(%rsi)   // new_context.resumable = 0
           ^
<inline asm>:13:13: error: Expected ,, instead got: (
      movq 0(%rsi), %rsp  // %rsp = new_context.stack_top
            ^
<inline asm>:14:12: error: Unexpected token in operand: %
      popq %r15         // pop callee-saved registers from the stack
           ^
<inline asm>:15:12: error: Unexpected token in operand: %
      popq %r14
           ^
<inline asm>:16:12: error: Unexpected token in operand: %
      popq %r13
           ^
<inline asm>:17:12: error: Unexpected token in operand: %
      popq %r12
           ^
<inline asm>:18:12: error: Unexpected token in operand: %
      popq %rbp
           ^
<inline asm>:19:12: error: Unexpected token in operand: %
      popq %rbx
           ^
<inline asm>:20:12: error: Unexpected token in operand: %
      popq %rdi         // pop 1st argument (for initial resume)
           ^
LLVM ERROR: Cannot select: intrinsic %llvm.x86.sse2.pause
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.	Program arguments: ./llc hello.ll -march=wasm32
1.	Running pass 'Function Pass Manager' on module 'hello.ll'.
2.	Running pass 'WebAssembly Instruction Selection' on function '@"*Intrinsics::pause:Nil"'
 #0 0x000000000290da2b llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (./llc+0x290da2b)
 #1 0x000000000290dae2 PrintStackTraceSignalHandler(void*) (./llc+0x290dae2)
 #2 0x000000000290baa9 llvm::sys::RunSignalHandlers() (./llc+0x290baa9)
 #3 0x000000000290d473 SignalHandler(int) (./llc+0x290d473)
 #4 0x00007f36e3d6cb20 __restore_rt (/lib64/libpthread.so.0+0x12b20)
 #5 0x00007f36e2c937ff raise (/lib64/libc.so.6+0x377ff)
 #6 0x00007f36e2c7dc35 abort (/lib64/libc.so.6+0x21c35)
 #7 0x000000000285d671 llvm::install_bad_alloc_error_handler(void (*)(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool), void*) (./llc+0x285d671)
 #8 0x000000000285d4b5 llvm::report_fatal_error(llvm::StringRef, bool) (./llc+0x285d4b5)
 #9 0x00000000026549f5 std::remove_reference<llvm::SelectionDAGISel::UpdateChains(llvm::SDNode*, llvm::SDValue, llvm::SmallVectorImpl<llvm::SDNode*>&, bool)::'lambda'(llvm::SDNode*, llvm::SDNode*)&>::type&& std::move<llvm::SelectionDAGISel::UpdateChains(llvm::SDNode*, llvm::SDValue, llvm::SmallVectorImpl<llvm::SDNode*>&, bool)::'lambda'(llvm::SDNode*, llvm::SDNode*)&>(llvm::SelectionDAGISel::UpdateChains(llvm::SDNode*, llvm::SDValue, llvm::SmallVectorImpl<llvm::SDNode*>&, bool)::'lambda'(llvm::SDNode*, llvm::SDNode*)&) (./llc+0x26549f5)
#10 0x00000000026541a1 llvm::SelectionDAGISel::SelectCodeCommon(llvm::SDNode*, unsigned char const*, unsigned int) (./llc+0x26541a1)
#11 0x000000000137adca (anonymous namespace)::WebAssemblyDAGToDAGISel::SelectCode(llvm::SDNode*) (./llc+0x137adca)
#12 0x000000000137c6aa (anonymous namespace)::WebAssemblyDAGToDAGISel::Select(llvm::SDNode*) (./llc+0x137c6aa)
#13 0x0000000002646c81 llvm::SelectionDAGISel::DoInstructionSelection() (./llc+0x2646c81)
#14 0x000000000264606f llvm::SelectionDAGISel::CodeGenAndEmitDAG() (./llc+0x264606f)
#15 0x0000000002644756 llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction, true, false, void>, false, true>, llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction, true, false, void>, false, true>, bool&) (./llc+0x2644756)
#16 0x00000000026491c3 llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) (./llc+0x26491c3)
#17 0x00000000026433d9 llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) (./llc+0x26433d9)
#18 0x000000000137ad95 (anonymous namespace)::WebAssemblyDAGToDAGISel::runOnMachineFunction(llvm::MachineFunction&) (./llc+0x137ad95)
#19 0x0000000001a2d892 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (./llc+0x1a2d892)
#20 0x0000000001fd1dbd llvm::FPPassManager::runOnFunction(llvm::Function&) (./llc+0x1fd1dbd)
#21 0x0000000001fd2012 llvm::FPPassManager::runOnModule(llvm::Module&) (./llc+0x1fd2012)
#22 0x0000000001fd23fa (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) (./llc+0x1fd23fa)
#23 0x0000000001fcdbd5 llvm::legacy::PassManagerImpl::run(llvm::Module&) (./llc+0x1fcdbd5)
#24 0x0000000001fd2bd5 llvm::legacy::PassManager::run(llvm::Module&) (./llc+0x1fd2bd5)
#25 0x00000000012d346a compileModule(char**, llvm::LLVMContext&) (./llc+0x12d346a)
#26 0x00000000012d12a9 main (./llc+0x12d12a9)
#27 0x00007f36e2c7f7b3 __libc_start_main (/lib64/libc.so.6+0x237b3)
#28 0x00000000012d036e _start (./llc+0x12d036e)
Aborted (core dumped)


$cat /etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
1 Like
14

That’s awesome. Looks like you’re on the right track here. I’m not sure what the error is, but it’s a start at least!

15

Yeah, compiling to WASM target is technically not an issue with LLVM. The problems come with embedding Crystal’s stdlib runtime into the WASM runtime. The shown error is caused by the concurrency primitives. Stack context swap is implemented in inline assembly and architecture specific.
Hooking up garbage collection is also tricky IIRC.

That being said, a simple Crystal program without stdlib (--prelude=empty) should be able to compile to WASM just file.

16

I saw the simliar explanation at Crystal-lang: why is the LLVM "hello.bc" not the same if generated by Crystal or by clang? - Stack Overflow

17

Thanks! As you said, I do these two trys:

$ cat hello4.cr ## from https://stackoverflow.com/questions/52877191/crystal-lang-why-is-the-llvm-hello-bc-not-the-same-if-generated-by-crystal-or
require "lib_c"
require "c/stdio"

LibC.printf pointerof("hello world".@c)

$ crystal  build hello4.cr  --emit llvm-ir  --prelude=empty # get *.ll
$ ./hello4
hello world
$ date
Tue Mar  2 21:59:15 EST 2021
$ emcc hello4.ll  -s EXIT_RUNTIME=1 -o hello4.html # output hello4.wasm and hello4.js and hello4.html
warning: overriding the module target triple with wasm32-unknown-emscripten [-Woverride-module]
1 warning generated.
$ ls -l hello4.*
-rw-rw-r-- 1 myth myth 102676 Mar  2 21:59 hello4.html
-rw-rw-r-- 1 myth myth 121687 Mar  2 21:59 hello4.js
-rwxrwxr-x 1 myth myth  12284 Mar  2 21:59 hello4.wasm
-rw-rw-r-- 1 myth myth   5133 Mar  2 21:59 hello4.ll
-rw-rw-r-- 1 myth myth     76 Mar  2 21:32 hello4.cr

$emcc -v
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 2.0.14
clang version 13.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project 5f3c99085d4c2ebf57fd0586b013b02e32a8e20b)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/myth/emsdk/upstream/bin

$crystal --version
Crystal 0.36.1 [c3a3c1823] (2021-02-02)

LLVM: 10.0.0
Default target: x86_64-unknown-linux-gnu

and

$ cat hello2.cr # hello2.cr file is  empty ( 0 line)
$ crystal  build hello2.cr  --emit llvm-ir  --prelude=empty # get *.ll
$ ./hello2

$ date
Tue Mar  2 21:58:56 EST 2021
$ emcc hello2.ll -o hello2.html # output hello2.wasm and hello2.js and hello2.html
$ ls -l hello2.*
-rw-rw-r-- 1 myth myth 102676 Mar  2 21:58 hello2.html
-rw-rw-r-- 1 myth myth 119744 Mar  2 21:58 hello2.js
-rwxrwxr-x 1 myth myth   1385 Mar  2 21:58 hello2.wasm
-rw-rw-r-- 1 myth myth   4837 Mar  2 21:58 hello2.ll
-rw-rw-r-- 1 myth myth  15482 Mar  2 21:28 hello2.s
-rw-rw-r-- 1 myth myth   2956 Mar  2 21:24 hello2.o
-rw-r--r-- 1 myth myth   3756 Mar  2 21:09 hello2.bc
-rw-rw-r-- 1 myth myth      0 Mar  2 21:09 hello2.cr

$emcc -v

$crystal --version
Crystal 0.36.1 [c3a3c1823] (2021-02-02)

LLVM: 10.0.0
Default target: x86_64-unknown-linux-gnu

Above two trys both get the *.wasm~

Start a http server and open the hello4.html with Chrome, it really output the “hello world” in the console log~ Thanks！

3 Likes
18

Crystal’s simple puts call has a much more behind it. It is based on libraries for handling asynchronous IO, concurrency, signal handling, garbage collection and more. Some of these libraries are completely implemented in the Crystal standard library, some use other libraries that are either directly embedded into the binary ( libgc ) or still require dynamic libraries from the system ( libpcre , libpthread )
Above is referenced from Crystal-lang: why is the LLVM "hello.bc" not the same if generated by Crystal or by clang? - Stack Overflow

So the main problem is how to compile the Crystal stdlib library and libgc/libpcre/libpthread to wasm?
Do we have something which be as reference for other program language resolve the similar problem?

19

If that can inspire:

This is to include WASM runtime.

1 Like
20

It goes a little bit beyond that, but that’s generally the case. I’ve investigated porting Crystal to wasm32-wasi a few times over the last year and been able to get some simple programs compiling and running via various WASM runtimes. I’ve documented some of this here: GitHub - maxfierke/crystal-wasm-tools: Tools for compiling Crystal dependencies, bindings, etc. for use with wasm. I had to disable a lot of features within the stdlib to get anything working, so it’s very very rough, and not super usable for anything particularly exciting

libpcre is relatively easy to port over (documentation contained above), but libgc and libevent2 present difficulties (pthreads are a non-starter until there’s an accepted WebAssembly primitive for threads)

The big blockers are:

  • Fiber support. As @straight-shoota touched on, context switching is impossible to do the way we do for other targets, as WebAssembly does not use registers and doesn’t have anything yet for creating separate stacks. There is a feature in binaryen called asyncify, that could theoretically be used and I started experimenting with but couldn’t get it to work, in order to implement context switching for Fibers. This would make libevent2 somewhat possible for evented I/O, but I had to hack around quite a lot to get libevent2 to even compile for wasm32-wasi, and it basically required taking a hatchet to anything involving networking or socket APIs, since those don’t exist in WASI for the most part
  • Exceptions. Exception handling is not something that exists in WASM yet with WASI, and so there’s no support for C++ exceptions in LLVM and WASM, except for some experimental stuff at the IR layer, but nothing that’s really usable or stable. This means that any exception in Crystal would just crash the program, and trying to avoid exceptions completely is difficult and makes it hard to write anything useful or use much of the stdlib
  • Require ordering weirdness. I found when I started disabling parts of the stdlib, it affected require load ordering and led to a rabbit hole of fixing ordering dependencies in the stdlib, so not sure if there’s a bug there or if it was really something related to the WASM work, but that made it pretty unpleasant as well

It’s possible that targetting wasm32-emscripten, instead of wasm32-wasi, would yield better results, but I didn’t want to have to figure out how to shoehorn emscripten’s custom toolchain wrappers into compiling Crystal and dealing with JavaScript wrappers, etc.

In general, the big blockers really lie at the WebAssembly or LLVM-level, and those proposals move somewhat slowly, so I would caution anyone holding out hope for Crystal on WebAssembly that it will be a while unless the runtimes or toolchains can provide more native capabilities for Crystal to build on

11 Likes