RFC: `with ... yield` replacement


#21

I think we are getting caught in conflation of two separate issues:

  • DSL implementation
  • Procs with no explicit receiver

I think DSLs in Crystal are better served with macros, which are even more opaque.
Receiverless procs are useful outside of DSLs as well, but the core team should make a judgement whether they want this language feature in the first place.

Also getting back to App.routes example why not just write:

class App
  get :foo
  post :bar
end

It’s not like you are doing it in runtime anyway.


#22

For me, DSL is about defining the valid grammar. For that you might create some helper types that will have a narrow interface, limiting what methods can be typed.

App.routes& do
  get :foo
end

could work if App.routes(&block : App::DSL::Routing -> _) and there is a App::DSL::Routing#get.
That same thing in App.tls would be what limit you for not calling #get. Because there is not.
Of course, you can call App.routes inside App.routes (unless a runtime check like the one to detect nested its is done).

Some scenarios work fine with macros. Definitely, but is harder to limit the grammar.

It’s all about the grammar. Sometimes code that uses DSL can be used to produce data or maybe they can be executed multiple times in different context. Again it’s about the grammar.

The Kotlin alternative will work only for static methods I think (I might be wrong). So you are forced to pass an argument if you want some not empty/default context. Note that if only static methods are to be used then the problem goes a bit away because you will be able to track easier the definition.

Last, something that it seem to get ignored is that by having this explicit change of implicit receiver the compiler could (theoretically) treat the most majority usages of blocks body without needing the method body. That would be a win for modular inference & compiling #someday.


#23

Every time I think I understand the problem and every time I don’t. I feel really stupid, and I’m sorry for that.

Initially I though that App.routes& syntax would always rebind the block to App, but now I think it just indicates that the receiver would change to something. Is it really helpful though?

As I see it, the problem example would be multiple methods with compatible signatures defined in different places and a user who’s not sure which one gets called, right? This is not a problem exclusive to with yield though. Overloading, previous_def, macros and dynamic binding in general all get in the way of static analysis and “write time” reasoning. (Just a week ago I thought I was calling spawn method, but ended up calling spawn macro.)

Have you though about instead of marking the method as rebinding its block, marking the block as rebindable, and even rebindable with type restrictions? It’s not gonna be a clean syntax though, because this is a simple case:

App.routes do(App::DSL::Routing)
  get :foo
end

#24

It’s not about rebinding the block or anything. It’s about syntax sugar. Essentially what Brian wrote in the first post. This:

App.routes& do
  get :foo
  post :bar
end

Translates to this:

App.routes do |c|
  c.get :foo
  c.post :bar
end

There’s no more than that. If we go that route there’s no need to have with ... yield, you can just do yield some_object. In fact it could work even if the author that wrote the method didn’t think of it being used like that.

For example:

('a'..'z').each& { ::puts upcase } # print 'A', 'B', etc.

That will work, because the above is expanded to:

('a'..'z').each { |c| ::puts c.upcase }

The way it works is it just rewrites all methods that don’t have a receiver to method that have an a receiver that’s the first argument yielded to the block.

(I had to use ::puts instead of puts to mention the top-level because otherwise it will be rewritten as c.puts, which is not what I want)

In fact, if we go that route you could even capture those blocks to invoke them later:

class Foo
  def initialize(@target : String)
  end

  def foo
    puts "Hello #{@target}!"
  end
end

def callback_on_foo(&block : Foo ->)
  block
end

block = callback_on_foo& do |x|
  foo
end

block.call(Foo.new("world"))

I like that & is used for this, because in the previous example you have:

callback_on_foo& do |x|
  foo
end

which can also be written like this:

callback_on_foo &.foo

However, we can’t use the call& do syntax because & is a binary operator (call & exp) and we’ll need a look ahead to check if do or { comes after &. Maybe it’s fine, but I generally don’t like look aheads when parsing.

Also, what if the method has arguments, how do you write it?

call(1, 2)& do
end

call&(1, 2) do
end

It’s a bit strange. I do like & but maybe it should be near the block, because it’s essentially changing the block. So maybe:

call(1, 2) do&
  something
end

call(1, 2) {& something }

Or maybe even:

call(1, 2) do &.
  something
  something_else
end

call(1, 2) { &. something }

# With a real example
App.routes do &.
  get :foo
  post :bar
end

You put &. after do or { and it means “calls that don’t have a receiver now take the first argument as a receiver inside the block”… which is what we are using &. for right now.

And putting & after do or { in the beginning of a block gives a parsing error right now, so that syntax is available and it can be implemented without a look ahead.

Finally, doing it this way will simplify the compiler’s code, which is always good because the simpler it is, the easier it is to understand it and to evolve it.


#25

Thank you for in-depth explanation, Ary.

I have an idea.

Edit: nevermind.


#26

@asterite The idea to use &. looks appealing to me:

App.routes do &.
  get :foo
  post :bar
end

In call(1, 2) { &. something } however, isn’t it the same as call(1, 2) { &.something }?

This works now and prints "1" 2 times:

def foo
  yield 1
end

p foo &.to_s
p foo &. to_s

I know &.do shouldn’t be used, since it currently expands to { |x| x.do }. The &. do currently also expands to { |x| x.do }.

But if it wasn’t, we could also try to put &. before blocks?

App.routes &. do
  get :foo
  post :bar
end

call(1, 2) &. { something }

#27

Kotlin language uses it as implicit name for first argument to the block: strings.filter { it.length == 5 } which looks nice.

How about this crazy idea with less visual noise of & or &.? :)

App.routes doit
  get :foo
  post :bar
end

Or even:

App.routes it
  get :foo
  post :bar
end

There is no syntax for {} in that case, but is it really useful for single line blocks? I would expect the real DSL use case to favor multiline blocks anyways.


#28

I like using a different keyword, rather than syntax or a symbol, for this. We still have then pretty much free and I think it would read more nicely than it. It could be read as descending into the scope of the first block argument. And I wouldn’t even mind removing it from case when then to reduce ambiguity, I never needed it there. Alternatively we could also move the with syntax to the callee side like this:

with App.routes do 
  get :foo 
  post :bar 
  puts it
end

though I can see that might be a tad confusing being about App.routes first block argument rather than its return value as you’re used to from say Python.

EDIT: Thinking about it, if we give up postponed evaluation of the block and capturing for this feature, we could generalize and do make it the value of expr in with expr do, so a general call syntax for moving the receiver to expr for a given scope. I could see this making it more powerful and worst case for having to have lazy evaluation with this in a DSL would be

App.routes.draw do |builder|
  with builder do
    get :foo
    post :bar 
  end
end

Sorry if this came up before, didn’t read the whole discussion here and in GH tbh :blush:


#29

Agree. I was more concerned not about using some particular word, but removing noise. Also it is already taken by Spec, so we shouldn’t use it.

The with App.routes do looks really nice to me, also I think it can cause confusion, at least when seeing it first time and mostly because of this particular word with and it’s meaning in other languages.

OTOH it might actually be a quick thing to get accustomed to. Also finding other word for this case might be tricky.


#30

Should we also care about not only the first yielded argument but having the pass other arguments?:

with App.routes do |two, three|
  get :foo, two, three
end

Will be the same as:

App.routes do |it, two, three|
  it.get :foo, two, three
end

And yielder:

class App
  def routes
    yield it, two, three
  end
end

#31

Another idea.

What if we use some special name for first block argument to mark it as implicit receiver?

Would it make quite obvious from first look what’s going on here? :thinking:

App.routes do |it, two, three|
  get :foo, two, third
end

Or maybe…

App.routes do |&., two, three|
  get :foo, two, three
end

Or…

App.routes do |&, two, three|
  get :foo, two, three
end

Also works with

App.routes { |&, two, three|  get :foo, two, three }

Calling multiple methods on implicit receiver is nice (the &. doesn’t allow this):
[1, 2, 3].map {|&| "#{to_s} : "{to_s(16)}" }

Expands to:
[1, 2, 3].map {|n| "#{n.to_s} : "{n.to_s(16)}" }


#32

Reading all of the responses persuades me that it would be better to leave it as is. Almost any solution makes the code uglier. I think it’s OK to have such an implicit feature (with yield), because most of the time it’s used with DSL, and DSLs tend to be well-documented.


#33

I like the idea of |&|, it could even be possible to use it for not-the-first argument if the user of the dsl ever want that:

foo do |one, &, three|
  bar one, three
end

The only drawback I see would be that we loose the name of the argument along the way, but as we already don’t have it in this case maybe it’s not a problem


#34

@bew Not sure about good use case but we could keep argument name for explicit use. Something like:

foo do |one, two&, three|
  bar one, three
  baz one, two, three
end

Or this. Reminds of instance var, but probably “at two” kind of makes sense to me.

foo do |one, @two, three|
  bar one, three
  baz one, two, three
end
foo do |one, @, three|
  bar one, three
end

#35

How about just accepting self in block args?

setup_routes = ->(self, global_prefix) do
  get "#{global_prefix}/foo"
  post "#{global_prefix}/bar"
end

App1.routes("/app1") &setup_routes
App2.routes("/app2") &setup_routes

#36

The problem with with expr do is that the arguments of expr will need to have parenthesis.
So I would prefer to leave expr be top called method.

@asterite If we go with & it should be on method name to keep the parenthesis optional. Even if that is a lookahead in the parser since that is only in the method name, while parsing identifiers. It’s in a well defined place.

We need to support arguments without noise as much as possible to have a good DSL. Something as similar as:

div class: "foo" do
  p "bar"
end

The with for any object will be expr.tap ... with whatever syntax could be.

I haven’t used DSL where the yielded grammar block has many arguments. I would not mind leaving that out of scope and using arg.tap ... to mimic that.

Back to syntax. I agree using a new keyword looks cleaner. But we will lose the chance to use expr { }. I can accept that. It will also allow better error reporting. Let’s say we use doit, then if the block wants or the body yields other than 1 argument, then, there is a compile time error.

I also thought that does might play good as a keyword.

Let’s try the discourse polling to collect the opinions.

  • method& (supporting do/end and { })
  • do& / {&
  • Use doit instead of do (not supporting { })
  • Use does instead of do (no supporting { })
  • Change nothing

0 voters


#37

Where’s my &. proposal in that poll? :slight_smile:

@vlazar Yes, the intention is that foo { &. something } looks similar to what we have right now. In fact I wouldn’t mind foo { &.something } to work too. Or we could change it to be before the block like you propose (making &. a keyword of sort), so foo &.{ something }. I think I like your proposal the best, since:

foo &.something

can be seen as a shortcut to:

foo &.{ something }

with that &.{ ... } meaning "use the first block argument implicitly in the entire block.

Now, to use &. you can do:

foo &.something

but if you have arguments you have to write it like:

foo 1, 2, &.something

Notice the comma before &.. That’s because & is a block argument, it’s not a block, and arguments come separated by comma. So overloading &. might bring a lot of confusion, but I’m not sure.

With do it’ll be:

App.routes &. do
  get :foo 
  post :bar
end

App.routes 1, 2 &. do
  ...
end

App.routes(1, 2) &. do
  ...
end

But not this:

App.routes 1, 2, &.do
  ...
end

because &.do expands to block_arg.do… unless we consider that as an exception, but exceptions are bad.

So I still don’t know which one I prefer. Probably &. after do/{ is better because it can’t be confused with &. before the block.


#38

Actually, I’m thinking that having &.do and &.{ as special constructs might be the best because, as I said above, they can be seen as applying &. to the entire block. Now I don’t mind having &.do not mean block_arg.do.


#39

@asterite Have you noticed my other idea of having a special name for block argument like & to mean the same thing? Could it work?


#40

With clean DSL argument in mind above everything else… The doit or does are cleaner than using special characters but reads a bit weird and are long words.

I remember the feeling when I was learning Ruby first when everything was looking so nice, but some things like do ... end were just too noticeable in Ruby after less noisy { ... } in other languages. Get used to it eventually, but it took me quite some time.

So it’s time for more crazy Friday ideas! :wink:

Are there any good two letter English words we can use to convey the meaning? There are as, in and of already https://github.com/crystal-lang/crystal/wiki/Crystal-for-Rubyists#available-keywords

With some other candidates from https://en.oxforddictionaries.com/explore/two-letter-words/

word meaning
at expressing location or time
my belonging to me
on supported by or covering
to expressing direction in relation to a location

We can do:

div class: "foo" at
  p "bar"
end

div class: "foo" my
  p "bar"
end

div class: "foo" on
  p "bar"
end

div class: "foo" to
  p "bar"
end

And maybe add of if current meaning can be removed:

div class: "foo" of
  p "bar"
end

Not sure if any of these make sense, but the shorter the keyword the less noise we will have in DSL.

EDIT:
We can also use yo (roughly meaning “with yield … do”). The popularity would probably skyrocket! :joy:

div class: "foo" yo
  p "bar"
end