"Why isn't there an LSP for Crystal?"

Crystal Contrib Tooling
1

There was a recent post on Reddit asking why there wasn’t an LSP for Crystal. This is a copy / paste of what I said in response, repeated for visibility and further discussion.

There are a number of things that make writing a language server for Crystal difficult.

  • There is no fault tolerant parser, which is necessary to provide functionality with “broken” code (well, there was before the tree sitter got a lot of improvements, at 99.07% of the stdlib parsing without errors now)
  • Crystal’s complex syntax makes it incredibly difficult to make a fault tolerant parser
  • There is no semantic analysis done (by the compiler) to methods that aren’t used
  • A full semantic analysis of the stdlib and all files is required (due to the nature of the language) every time a change is made
  • Type inference means that the types of variables and parameters depends on how they’re used, requiring semantic analysis of everything to ensure types are accurately captured
  • Everything is in a global namespace, and analysis relies on one or multiple entrypoints that have to be specified explicitly

Crystalline has been around for a few years and it does the best for what it has, re-using a lot of the semantic analysis that the compiler uses. It works for some people, especially on small to medium sized projects. I’d recommend giving it a try to see how well it works for you.

I’ve been working on a language server since last June, very much far from done but here’s a list of what I have so far:

My current approach that some people may not like, is that to get LSP-like features, methods / parameters / variables need to be explicitly typed. It’s the only way to avoid needing a full semantic analysis to resolve types.

Having spent the past year and a half working on tooling surrounding Crystal, I’ve become extremely familiar with the limitations of the language, and how it’s seemingly designed to be as hard as possible to have good tooling for. I know a lot of people in the community may be dissatisfied or frustrated with the current state of tooling, but the limitations of Crystalline are the limitations of the language itself, and it’s very hard to work around / outside of those without some compromise.

Progress is being made, but it needs to be done carefully and will take time.

3

Even if you get fault tolerant parsing right, there’s no a lot you can do to offer responsive autocompletion. For that you need to type-check everything and that’s slow.
An alternative is to only offer suggestions that may be possible, but not all of them, which might be good enough and it might also be responsive.
For example, in this code (where | is where the cursor is):

def foo(x)
  x.|
end

we know nothing about x. But we still know it’s an Object so we could suggest every possible method in the system, or maybe in this particular case just methods defined in Object to shorten that list.
But then if you have this:

def foo(x)
  y = x.abs
  y.|
end

We don’t know what’s the type of x but we know it has an abs method. We could search all the types that have abs in them and know y is the return type of those… if the return type is explicit (though in this particular case we could type-check abs because it has no arguments). Then we offer methods of those types.

You kind of need to replicate the logic of the compiler here, where you “kind of” type-check a method with whatever information you have and proceed to the next expression, type-check it, etc., but in a different way than what the compiler does.

If we have a type annotation on an argument:

def foo(x: Foo)
  x.|
end

then we can use that to offer methods from Foo or any of its subclasses.

Given the type-less nature of Crystal, this is in my mind the correct LSP server to build for Crystal, not one that offers precise completion (because that’s impossible, and also very, very slow).

4

This is definitely acceptable, please just do it!

5

For sure. Simple heuristics can give you like 70% of experience of full intellisense and is definitely better than what we have right now. Like I don’t know what irb does, but it’s honestly pretty good, I wish I could get that experience in my ide.

6

I think I was originally approaching this of requiring types to get autocomplete, but I think you bring up a really great point I haven’t thought of, and a potentially different way of looking at this problem.

In the case of this:

def foo(a)
  b = a.|
end

I think all possible methods should be autocompleted, regardless of what foo is called with, if at all. Then whatever methods are called on a, not only restrict the type of b based on the return type of the methods, but also restrict the type of a based on what objects have those methods. So:

def foo(a)
  b = a.abs
  # Let's pretend this restricts `a` to `Number`
  a.|
  # ^ autocomplete only methods on `Number`
  a.bar
  # ^^^ error: method `bar` doesn't exist on `Number`
end

I may go with the more strict need-to-type-things approach to start with, but this does provide a lot of promise for larimar and ameba to do partial / lossy semantic analysis (I intend to re-use the same visitor for both).

I wonder if it’d also be possible to bubble up parameter types from their bodies? Like in the same example:


def foo(a)
  b = a.abs
  # Let's pretend this restricts `a` to `Number`
end

foo("bar")
  # ^^^^^ error: parameter `a` should be `Number`, not `String`
  # or
  # ^^^^^ error: String does not have method `abs`

Something to explore once more a framework is laid out.

7

IMO explicitly typing everything is a minor inconvenience at worst and leads to safer code in the end, so it’s 100% worth it.

Type information on hover is the feature I personally would like the most, for which strict typing is certainly a prerequisite.

8

I’ve been using GitHub - elbywan/crystalline: A Language Server Protocol implementation for Crystal. 🔮 for a while now.

The only issue I’ve had with it is that it slows down over time… I don’t spend much time programming Crystal to invest in debugging this; but it works for my other LSP needs.

If you’re on Reddit — I’m not — please share this there :pray:

9

At first, this made me sad. Then I thought a bit more about it. In practice, it’s probably quite fine, as one tends to type much anyway.

But this would be extra cool.

I’m not sure I agree. My method might only require the argument you pass it answer to [](String) with a String, but if I typehint it to Hash(String, String) I limit your options on what you can pass. No passing it a HTTP::Headers, no matter how convenient it might be.

10

I am not familiar with LSP, but something just chocked me about Crystal:

Crystal’s complex syntax

??? I would like to understand. Have you got an example ?

11

I think that’s a bit of a special case. In quite many cases it’s entirely possible to easily type most parameters.
And I think that’s generally a good idea because it documents the interface of a method.

Also we can improve cases like the one you described to make it possible to type them in a meaningful way. For example Add a module for implementing Hash-like objects · Issue #10886 · crystal-lang/crystal · GitHub might be useful for that. Another approach might be Constrain type vars / free variables to specific types · Issue #934 · crystal-lang/crystal · GitHub with an implicit interface declaration.

12

Well, it’s duck typing. And it’s the cases left over from “quite many cases” that I’m thinking about.

My personal favourite is Go style interfaces. Move focus from what a thing is to how you interact with it.

13
14

Well, maybe in 2.0 then… Anyway, I just wanted to point out that not everybody thinks “just type everything” is all it’s cracked up to be, but I won’t derail the threat further.

But an LSP that can complete on variables with known types, and make an educated guess for those without, would be excellent. Is larimar in a state where it’s worth trying out, or is it still to early?

15

That is true. There is a middle ground of something closer to type hints rather than type restrictions (see crystal-lang/crystal#15334) that I’ve also been thinking about.

Things are still too early, the only things that exist currently are semantic tokens using the tree sitter and ameba integration. I’ll let people know when there’s something that’s worth testing.

16

The Ruby LSP has become really mature recently. I would recommend taking a look at it and trying it with their VSCode extension: Ruby LSP | An opinionated language server for Ruby. Batteries included!

17

This may be a silly idea but:

If having type information everywhere would make the LSP job’s easier … why not add it in a “shadow” source tree and use that in the LSP?

For example, using GitHub - Vici37/cr-source-typer: Adds type restrictions to crystal source code and instead of writing the type information into the source files just keeping it in memory and use that as extra information to help the LSP?

18

Good tool. I wish the crystal compiler had a way to force me to use types instead of having to work with both static and dynamic typing. I personally think that Crystal team should have decided between using static or dynamic typing at the time of creating the language, not just both and let the compiler do type inference on the variables, but that decision was already done years ago.

I tried this for a project a few days ago and it works really great.

19

That’s an idea I’ve messed around with a little, that or something akin to a compilation database. There are a few issues though, primarily that those types are the types informed by their use, and not by all possible allowed values. I have some other ideas that I’m playing around with that should be more flexible / faster.

20

Good tool. I wish the crystal compiler had a way to force me to use types instead of having to work with both static and dynamic typing. I personally think that Crystal team should have decided between using static or dynamic typing at the time of creating the language, not just both and let the compiler do type inference on the variables, but that decision was already done years ago.

There are some ameba rules to be released at some point that enforce adding type restrictions to method params / return among others, which should help with what you’re looking for.

It’s all static typing, and even when providing a type restriction, it’s just a restriction and doesn’t necessarily reflect the actual type at the end of semantic analysis. The compiler is free to reduce a type restriction - from say String | MyClass? - to String, MyClass, Nil, or MySubClass all depending on how the parameters and such they’re attached to are used.

21

Thanks for the shout-out! This was exactly going to be a followup side project of mine, but mass-applying types to the standard library and life in general has gotten in the way :grimacing:

Updating the existing tool, or creating a new one using the source type as inspiration, to spit out a json / sqlite / file-based representation of the inferred types would be straight forward. Another side-side-project was to experiment with what a static code analyser would look like. Similar to Ameba, but you’d be able to get more in depth with the checks you were trying to make.

But this thread is about LSPs :)