[RFC] Annotations metadata declaration/DSL

I recall that when annotations were first announced, the bodies of the annotation declaration were intentionally left blank for future expansion. Seeing as they’ve become much more useful and prevalent since then (JSON::Serializable, Athena routing, etc.), I think now would be a great time to start adding some more compiler verification metadata to them.

Judging from the discussion in this issue and from how other languages handle annotations, I think there are two main candidates for metadata:

  • Target (class, method, lib, etc.)
  • Parameters

This does raise the question of syntax. Should Crystal go down the path of Java with its meta-annotations or should it start doing compiler DSLs (á la Kotlin contracts)?

# meta-annotation
@[Target(:class, :method)]
annotation Foo
  str : String
  num : Int32 = 10
end

# DSL
annotation Foo
  target :class, :method
  
  param str : String
  param num : Int32 = 10
end

(Maybe types need to be specified as their ASTNode cousins: StringStringLiteral, Int32NumberLiteral?)

This might also open the door for processing logic to be put in the body as another macro:

annotation Bar
  target :ivar

  # possible syntax for unnamed parameters?
  params(TypeNode)

  macro annotated(target)
    {% if target.type != nil %}
      def {{ target.name }}: {{ params[0] }}
        # ...
      end
    {% end %}
  end
end
1 Like

Personally I love the DSL route, I also like the idea of handling all annotation logic inside of the annotation itself with something like that annotated macro. It just makes more sense than having to write a separate macro outside of the annotation that looks for annotations.

I kind of like how PHP handles it

/**
 * @Annotation
 * @Target({"PROPERTY","METHOD"})
 */
final class SomeAnn
{
    /**
     * @var string
     */
    public $name;

    public function __construct(array $values)
    {
        if (!isset($values['value']) || !\is_string($values['value'])) {
            throw new RuntimeException(sprintf('"value" must be a string.'));
        }

        $this->name = $values['value'];
    }
}

Or the crystal version would be like

annotation SomeAnn

  getter name : String

  # Initialize is optional, gets passed a Tuple and NamedTuple supplied
  def initialize(positional_args, named_args)
    # Can do some custom stuff here.  Otherwise properties must be defined as named arguments.
  ned
end

One good way I came up with to handle annotations currently is to define a class/struct that represents the annotation. Then you can iterate over the annotations doing like MyAnnClass.new {{ann.named_args.double_splat}}; which handles required properties, and default values etc.

Excessive annotation syntax infiltrating the language will slowly diminish Crystal’s beautiful syntax.

4 Likes

I think we could declare it like annotation Foo(args) because when you use it it’s exactly like a call, minus the block. For example:

module JSON
  annotation Field(*, key = nil, ignore = false, root = nil, converter = nil, presence = false, emit_null = false)
end

Type restrictions should be HashLiteral, StringLiteral, etc., because annotations operate on AST nodes. However, I’m not sure how useful this is because it’s hard to specify the types of keys and values. Or for example converter can be a type, a method, etc.

Then I don’t know about the annotated hook, we’d have to think of real use cases.

At the very least, I think it would be really helpful for documentation. I recently implemented DB::Serializable for crystal-db and I noticed that the only way to really describe the possible parameters for the annotation is through manually listing them inside a doc comment. Personally, I think it would’ve been much cleaner to do something like

module DB
  annotation Field(ignore : BoolLiteral = false, key : StringLiteral | NilLiteral = nil, converter : TypeNode | NilLiteral = nil)
end

and have crystal doc create some kind of documentation for it. At that point, I see no reason not to typecheck.

2 Likes

Bumping this. For what I do this is something I struggle with quite often. It’s quite annoying when you want to use an annotation for something, but then need to also have another type to represent the data that should be read off the annotation. Making the type that gets applied AND the type that stores/handles the data, the same type would be :100:.

IMO this would be the most common use case, I still think ideally annotations should work more like:

@[Annotation]
struct MyAnn
end

as opposed to:

annotation MyAnn
end

Essentially this would allow that type (either a class or struct) to be applied via the @[...] syntax. In macro land the type of MyAnn would be Annotation like it is now to allow reading the arguments off the annotation. But, a new method could also be added, like build or something, that would create an instance of the type based on arguments defined as part of the annotation. For example:

@[Annotation]
record MyAnn, name : String, active : Bool = true

@[MyAnn(name: "foo")
def foo
  {{ @def.annotation(MyAnn)[:name] }}
end

foo # => "foo"

@[MyAnn(active: false)
def bar 
  "bar"
end

bar # => Compile error because `name` wasn't supplied and it's not nilable nor has a default value

@[MyAnn(name: "baz")]
def baz
  {{ @def.annotation(MyAnn).build }}
end

baz # => MyAnn(@name="baz", @active=false)

IMO this would be a game changer as it would make annotations less unique, and more like standard types, supporting modules/inheritance etc.

I made an issue about this: https://github.com/crystal-lang/crystal/issues/9802.

A use case for macro annotated, precisely, is eliminating the need for additional DSL powered by the language itself. For example, we could simply write:

annotation Flags
  macro annotated(target, *args, **opts)
    {% unless target.is_a?(EnumDef) %}
      {% raise "can only apply @[Flags] to enums" %}
    {% end %}
  end
end

Then we could allow annotations to include or use an implementation for target, taking advantage of the fact that multiple hooks are all run and never override each other (so that we could still have our own annotated hook):

module AnnotationHelper
  macro target(*kinds)
    macro annotated(target, *args, **opts)
      \{% kinds = {{ kinds }} %}
      \{% if target.is_a?(ClassDef) && !kinds.includes?(:class) %}
        \{% raise "expected one of: #{kinds.map(&.id).join(", ").id}, got ClassDef but no :class" %}
      \{% elsif target.is_a?(EnumDef) && !kinds.includes?(:enum) %}
        \{% raise "expected one of: #{kinds.map(&.id).join(", ").id}, got EnumDef but no :enum" %}
      \{% elsif target.is_a?(Def) && !kinds.includes?(:method) %}
        \{% raise "expected one of: #{kinds.map(&.id).join(", ").id}, got Def but no :method" %}
      \{% end %}
    end
  end
end

annotation Flags
  include AnnotationHelper
  target :enum

  # less terse, compiler still needs to permit macro expansions in annotation scope
  AnnotationHelper.target :enum
end

The contents of the annotated hook should be interpolated after target in the same scope. One of my use cases is to automatically generate overloads to circumvent Crystal’s current overload ordering behaviour:

# :nodoc:
annotation ExpandPrecRnd
end

module Math
  @[ExpandPrecRnd]
  def sqrt(value : BigFloatR, *, precision : Int = BigFloatR.default_precision, mode : BigFloatR::RoundingMode = BigFloatR::RoundingMode.default)
    BigFloatR.new(precision: precision) { |mpfr| LibMPFR.sqrt(mpfr, value, mode) }
  end

  {% for a_def in @type.methods %}
    {% if a_def.annotation(ExpandPrecRnd) %}
      def {{ a_def.name }}({{ a_def.args[0...a_def.splat_index].splat }})
        {{ a_def.name }}(
          {{ a_def.args[0...a_def.splat_index].map(&.name).splat }},
          precision: BigFloatR.default_precision,
          mode: BigFloatR::RoundingMode.default,
        )
      end
    {% end %}
  {% end %}

  # the above generates:
  # ```
  # def sqrt(value : BigFloatR)
  #   sqrt(value,
  #     precision: BigFloatR.default_precision,
  #     mode: BigFloatR::RoundingMode.default,
  #   )
  # end
  # ```
  # because the original def is _less_ specific than the `def sqrt(value)`
  # overload in stdlib (the one without parameter restrictions)
end

This approach is rather imperative, as that inline macro must be copied in every scope that uses ExpandPrecRand. An annotated hook would make this declarative:

annotation ExpandPrecRnd
  macro annotated(target, *args, **opts)
    # extra error handling (e.g. `args.empty?`, `target.is_a?(Def)`)

    def {{ target.name }}({{ target.args[0...target.splat_index].splat }})
      {{ target.name }}(
        {{ target.args[0...target.splat_index].map(&.name).splat }},
        precision: BigFloatR.default_precision,
        mode: BigFloatR::RoundingMode.default,
      )
    end
  end
end

module Math
  @[ExpandPrecRnd]
  def sqrt(value : BigFloatR, *, precision : Int = BigFloatR.default_precision, mode : BigFloatR::RoundingMode = BigFloatR::RoundingMode.default)
    BigFloatR.new(precision: precision) { |mpfr| LibMPFR.sqrt(mpfr, value, mode) }
  end

  # automatically generates the same def as the previous snippet
end

I am not a fan of declaring the annotation parameters right next to the annotation name, since to me that implies each annotation may only have one “overload”. Likewise, I don’t think we should impose a one-to-one correspondence from annotations to “real” Crystal types; this prevents users from mapping different annotations to the same type for e.g. performance reasons, but also this correspondence can probably be implemented in 100% user code if we allow user macros in annotations instead.

Slightly related to the last point is that you can already define methods on annotation metaclasses:

annotation A
end

def A.foo(x, y)
  x + y
end

A.foo(1, 2) # => 3
6 Likes

PHP allows inheriting their annotation types which makes it super easy to define more specialized sub-annotations for example. This would also be great, and maybe a reason for them to be class and not struct given you can’t inherit non-abstract structs?

[RFC] Annotations 2.0 · Issue #9802 · crystal-lang/crystal · GitHub did a lot of exploration/thinking about how annotations can be improved, with a working PoC if anyone is interested :slightly_smiling_face:

Maybe not strictly a comment on how to structure annotations, but the hardest part for me when I tried using my own annotations was how to find the annotated classes/methods during compilation. So something on my wish list would be some better docs (+ maybe some macros) for achieving stuff like:

  • finding all types in the current namespace (module) annotated with annotation X
  • finding all methods in the current namespace (module) annotated with annotation Y
3 Likes

Opened RFC: Improved Annotations by Blacksmoke16 · Pull Request #17 · crystal-lang/rfcs · GitHub for more focused feedback on my proposal.


Yea would def be a good point to re-vamp the docs/add some more examples. Esp the API docs. I think there should also be some new page/guide/something in the crystal book that serves as an intro to the macro API outside of a macro definition. I.e. better explains the macro AST setup and how to do common things like this.

Here’s a bit of a brain dump for future reference/inclusion in docs:

Are you saying you have something like:

annotation MyAnn; end

module MyApp
  module Foo
    @[MyAnn]
    record One
    record Two

    @[MyAnn]
    record Three
  end

  module Bar
    record Three

    @[MyAnn]
    record Four

    record Five
  end
end

and you’re wanting to only select the types within MyApp::Foo with a specific annotation? So [One, Three] in this example?

I’m not sure of a great way to do namespace filtering. Like you can do {% MyApp.constants %} and get [Foo, Bar] but they’re macro ID’s and can’t really do much with them. Could be nice to expose child namespaces that give you TypeNode instances. Something a bit more robust than like:

MyApp.constants.map { |c| parse_type("MyApp::#{c}").resolve }

Normally, at least how I been using annotations, is you start off by casting a wide net and then filter it down based on other criteria. Some examples of this are like:

  • All classes/structs
  • All subclasses of some specific type
  • All types that include a specific module
  • Some combination of the above

From here you could then go from “all types in the entire program” to “all types in the entire program with a specific annotation.” For example, getting all deprecated class/structs in the stdlib:

{%
 Object.all_subclasses.select(&.annotation(Deprecated)) # => [Atomic::Flag, Float::Printer::DiyFP]
%}

Once you have all your root data, you could then use additional #select calls to further filter the array down. So in this example could do something like:


{%
  Object
    .all_subclasses                            # Iterate thru all class/struct types
    .select(&.annotation(MyAnn))               # Filter down to only those with `@[MyAnn]`
    .select(&.name.starts_with?("MyApp::Foo")) # Filter down to only those in the `MyApp::Foo` namespace
%}                                             # => [MyApp::Foo::One, MyApp::Foo::Three]

You could then do something similar and iterate over each of the #methods in each type to do more filtering to get down to only the Def instances you care about.

@Blacksmoke16 - I think you are describing what I’m after very well. The solution I used look pretty similar to what you wrote…the thing that I feels “bad” to me is that we need to start at the very top (Object.all_subclasses) even if we know that the annotations only exists in types in the current namespace (or sub namespace).

I usually stay away from annotations because abstract modules can do the job of bringing unrelated types together quite well, but in this particular case I wanted to implement JSON-RPC 2.0 style routing to my project. Annotations seemed like the perfect fit for picking up the correct callback methods, but I was surprised how messy the macro code that built the “routing table” looked. It all works though, I just wanted to point out that the docs are how to do these kinds of things are a bit sparse.

1 Like