[RFC] Standardized Abstractions

Crystal Contrib
1

Standardized Abstractions

Extracting URI::Params is a struct? Why? - #10 by Blacksmoke16 to its own topic.

Background

Using decorators is usually a better approach when you want to customize/extend a type, especially for ones you do not own. For example:

struct CustomRequest
  getter request : HTTP::Request

  forward_missing_to @request

  def self.new(method : String, path : String, headers : HTTP::Headers? = nil, body : String | Bytes | IO | Nil = nil, version : String = "HTTP/1.1") : self
    new HTTP::Request.new method, path, headers, body, version
  end

  def initialize(@request : HTTP::Request); end

  def safe? : Bool
    @request.method.in? "GET", "HEAD", "OPTIONS", "TRACE"
  end
end

This way you can do something before/after a specific method, or add more methods, all the while not touching anything related to the upstream type itself.

HOWEVER, with the way the stdlib is implemented at the moment, this doesn’t work as well as it could since everything is typed as concrete implementations instead of abstractions. This means you cannot have this custom request type work as a drop in replacement to HTTP::Request , even if it is 100% compatible with it.

PHP’s PSR

PHP handles this via PHP Standards Recommendations - PHP-FIG, which define interfaces/requirements for various common libraries/implementations, such as eventing, caching, and HTTP clients, factories, and messages. This allows framework/application developers to design things such that any implementation could be used. E.g. an HTTP Client from Symfony, Guzzle, or even a custom implementation.

Ultimately this makes the user happier given they can use what they want and makes the developer’s life easier as they do not need to make a bunch of adapters or try and make things work with an arbitrary amount of implementations.

Composer’s provide

PHP’s package manager (composer) also taps into this feature via its provide key, which essentially allows defining a dependency like psr/http-client-implementation that would be satisfied if a library that is installed “provides” that package. E.g. one from the list of Packagist.

Proposal

I propose to make the stdlib less coupled with its concrete implementations by doing something similar to PHP’s PSR by introducing some interfaces (modules) that describe the public API that some common types must implement. This will allow using decorators as a drop in replacement, assuming they correctly implement the interface. E.g. include RequestInterface.

There are a few choices in regards to the exact implementation, which can be hashed out later.

Benefits

  • Stdlib better follows the SOLID principles
  • Provides a standard for shard creators and consumers
    • I.e. imagine using crest or halite interchangeably with HTTP::Client. Such as within OAuth2::AccessToken#authenticate among other places
  • Provides a place to define/document requirements/suggestions for implementers, such as that HTTP headers MUST be case-insensitive

Challenges

  • Overhead of determining the API and updating all usages in stdlib
  • How to distribute/provide the interfaces
  • Possibly somewhat breaking given not every method on HTTP::Request would be on the interface most likely
  • Not everyone may be familiar with SOLID, interfaces, or benefits thereof
9 Likes
2

I fully agree both on the proposal, it’s benefits and on the challenges it brings.

My opinion is that the benefits of such a good move easily overcome the challenges.

What is important is to build an easy path to help those who may be not familiar with these concepts, like SOLID, interfaces and etc.

In other words, the secret sauce is how we make this, with focus on make it understandable, with good documentation and, most important, for humans.

2 Likes
3

I really like Dart’s implicit interfaces for this. Downsides compared to your proposal of course are adding language and compiler complexity as well as feasibility of something like that with an open type system.

2 Likes
4

Oh, that’s an interesting concept. So you can extract the interface from any type on the fly. And implementing types are accepted in the same way as the original type.
This could be useful for its flexibility. It works without extracting an abstract interface, which would require cooperation from the owner of that type as well as consumers.

5

Also Structural subtyping can be considered, where names doesn’t matter. A concrete type is a member of a structural category if it happens to have all the needed elements, no matter what it happens to be called.

Structural typing is very similar to duck typing from dynamic programming languages, except that type compatibility is determined at compile time rather than at run time. If you’ve used Go, you’ll recognize that Go interfaces are structural types.

Major benefit is You do not declare structural relationships ahead of time, instead it is done by checking if a given concrete type can fulfill the required interface.

Just my two cents :smile:

5 Likes
6

This is pretty neat. But yea, is a lot to think about, like if someone monkey patches something in, what happens? I’m also not a huge fan that it would add everything to the interface. I.e. normally the type adds more helper methods and such than the base interface. This could also be an issue if/when more methods are added to the type, say HTTP::Client for example, as that would mean every implementation of it would need to add them?

This can work in some cases, but falls down when you want to have an ivar/class var store an instance of some interface, which you need the interface for to type it.

Ultimately I still think a decoupled dedicated interface is the most robust way forward. Just because it would allow refactoring or renaming of the core stdlib types without affecting the other users/implementers. If it was released as its own package(s), would also allow versioning the interface separately from the core types, which could be useful as, like everything else, these interfaces do not need to be static and can evolve over time.

7

They only inherit public API of course. Then I think typical usecases for this would be a kind of decorator where you would wrap an instance of the original type anyhow, so implementing all those methods could usually mean just forwarding them all to the original. Using macros or compiler aware forward_missing_to that then could be fairly low on boilerplate.

8

Yea I think that would be perfect for a decorator, as to your point, you’d essentially want a clone of that type’s API. But I’m still hesitant that it would be the best option for general abstractions, as the core idea of providing them is that you could use any implementation, not just those based on the core stdlib type, which would include additional helper methods which probably don’t need to be on the core interface.

Maybe we could do both? Something something helper macro/type/syntax for decorating a type via its implicit interface, while still providing explicit ones to handle the other use cases? Granted both of them do not have to be implemented at the same time.

9

Judging from the OP, none of this requires a change to the core language, right? How is this different from simply adding more modules / abstract classes?

1 Like
10

Exactly. The simplest solution to this is to come up with some modules, either in the stdlib or their as their own shard, that defines the API of some common types. Then include that module into the stdlib implementation, finally updating usages of the concrete stdlib type with that module (interface).

Anyone using say HTTP::Client would be backwards compatible as the type isn’t changing. But could/should update certain cases to use the interface, and library authors could update their libs to include it as well.

What needs to be hashed out still is like:

  1. What types do we want to do this with, and how far/specific we want to get
  2. What the API for each type should be
  3. How to distribute/provide the interfaces
  4. How to document/explain the reasoning for doing it

I’ll look more into some of the PSRs and come up with the Crystal equivalents, then can go from there.

2 Likes
11

Okay, so a fairly direct transaction of PSR-7 to Crystal would look something like: [RFC] Standardized Abstractions · GitHub.

The main change is that all their types are immutable, but not really sure I see the benefit there, so I just made everything be your standard getter/setters. They also have file uploads built directly into things, hence the UploadedFileInterface. Might not hurt to have a standardized way to represent uploaded files, even if the stdlib does not provide that ability by default.

There are also a few methods that are either new, or might be a bit tricker in Crystal, mainly on the ServerRequestInterface, specifically the #parsed_body and #attributes methods. Probably don’t need to ultimately have those on ours.

They also have a URIInterface which basically represents the stdlib’s URI type. Can’t really think of a use case where you’d need/want to define your own, so could just skip that and use URI directly, or even go a bit lower and have getter/setters for each part instead of requiring a URI instance.

2 Likes
12

Overview

Following up after Abstracting out adapters - Dependency Injection brought up a related problem. I’d be happy to put together a PR for this, but first we need to make some decisions. Specifically:

  1. Do we also want to include implementation requirements/recommendations?
  2. Where should this live?
  3. What should the interfaces actually consist of? How specific do we want to get?

I’m probably a bit biased, but it feels to me doing what PHP does with its PSRs and Java with its JSRs could be a worthy investment. Especially while the language is young. It probably wouldn’t be too hard to use those as points of inspiration given some of them are based on existing RFCs, if applicable, in order to port them to Crystal; if that’s the route we want to go.

The next sections are kinda me just thinking out loud.

API

I read thru PSR-7 again, and I think we could probably simplify it quite a bit as compared to my 1:1 port I came up with in my last post. Mainly due to the fact that we already have some pretty standardized APIs around URI and IO types. We also don’t parse files automatically, nor expose superglobal variables like $_POST to access data. Tho defining standardized ways to represent/expose the data is prob still worth doing?

The other major difference is that their types are immutable. I kinda see why they did this, but also don’t think it’s worth changing to. Especially given that would be a large breaking change, but this could be a future conversation for 2.0.

Location

The most obvious choice is to include these interfaces in the stdlib within the HTTP (or HTTP::Request even) namespace. This keeps the specification and interface close to the code, but does have a few drawbacks. Specifically updates to the spec couldn’t really be made/published without releasing a new Crystal version. It also makes it harder to introduce specifications that do not directly related to a stdlib feature, such as caching or what have you.

Having a monorepo that includes all of the specifications could be sufficient. Tho it would have to be determined how to manage breaking changes in only specific specifications. A many-repo approach could be a good middle ground, possibly in a dedicated organization. Another consideration is how to make these available to the stdlib in a way that they could be updated without releasing a new Crystal version.

Options

I think providing interfaces along with some implementation suggestions would be a good and not much extra effort. So ultimately it comes down to where to put them. There are a few possible paths forward, each with their own pros and cons:

  • Keep everything within the stdlib, close to related types if possible
    • Pro: Removes need to vendor in an external repo with the stdlib
    • Con: Changes to the interfaces/docs will need to wait on a Crystal release
    • Pro: All projects inherently have access to the interfaces
    • Pro: Easiest to implement
    • Con: Unclear on how to handle breaking API changes
      • Have to go with Crystal version? Use some flag to opt into what version you want? Have two versions with an alias that’ll change in the future?
  • Keep everything within the stdlib, but in a dedicated CSR module
    • Those mentioned before
    • Pro: Easier to include interfaces not directly related to stdlib w/o polluting top level
    • Pro: Would be easier to break out into its own thing if ever desired
  • Have an external monorepo that contains all the interfaces/docs
    • Con: How to expose them to the stdlib?
    • Con: How to handle breaking API changes
      • A branch per PSR? Something like what we’d do if they were in the stdlib?
    • Pro: Can be released/updated independently from Crystal itself
  • Have dedicated repos for each, managed via a monorepo
    • Those mentioned before
    • Pro: CSRs can be released/versioned independently of each other
    • Con: More overhead required to manage the extra repos and possibly org

Proposal

After thinking about it for a while, I’m leaning towards having them bundled with the stdlib, in a dedicated module, that isn’t required by default outside of the things using the interfaces. I think this provides the best balance between ease of use, implementation, and ease of maintenance. I looked at all the PSR packages and after their initial 1.0 release, they aren’t touched too often. Thus it’s likely our APIs will be similar, making it being tied to a Crystal release a non-issue. Changes/updates to the documentation will probably be more common, but that would work the same as improvements the stdlib documentation.

Proposed HTTP API 

module MessageInterface
  abstract def version : String
  abstract def version=(version : String) : String
  
  # Just going to leverage existing headers type for this
  # But, then might make sense to have a dedicated headers interface too
  abstract def headers : HTTP::Headers
  abstract def headers=(headers : HTTP::Headers)
  
  abstract def body : IO?
  abstract def body=(body : String)
  abstract def body=(body : Bytes)
  abstract def body=(body : IO)
  abstract def body=(body : Nil)
end

module RequestInterface
  include MessageInterface

  abstract def request_target : String
  abstract def request_target=(target : String) : String
  abstract def method : String
  abstract def method=(method : String)
  abstract def uri : URI
  abstract def uri=(uri : URI)
end

module ServerRequestInterface
  include RequestInterface

  abstract def cookies : HTTP::Cookies
  abstract def cookies=(cookies : HTTP::Cookies)

  abstract def query_params : URI::Params
  abstract def query_params=(query_params : URI::Params)

  # Optional
  #
  # Better than favoring monkey patching stuff into specific types.
  # Could use some sort of generic container to make it work, while still being type safe.
  abstract def attributes : _
  abstract def get_attribute(name : String) : _
  abstract def remove_attribute(name : String)
  abstract def attributes=(attributes : _)
end

module ResponseInterface
  include MessageInterface

  abstract def status : HTTP::Status
  abstract def status=(status : HTTP::Status)
end

It’s still a bit unclear on what to do when/if there is a breaking API change in a CSR. Mainly given they’re technically not tied to the Crystal’s semver versioning. Tho I also think we have options, and can figure it out later if ever needed.

Another thing that might be helpful, is also have a dedicated repo that could be use for extra CSRs, but before they’re “promoted” to the stdlib? But that’s also something we can hash out later.

1 Like
13

Can’t .headers be used instead?
Why is headers special? (compare with query_params or cookies)

14

Yes it definitely could. I left a comment in the code saying that’s also a possibility. They probably have these as dedicated methods because in PHP .headers would just return an assoc array. Given we have a dedicated type, probably be better to just leverage that.

EDIT: But might be worth having a dedicated headers interface too. I’ll see what that would look like. E.g. so I could have .headers return a ATH::Response::Headers and still be totally valid.

15

Another question that I thought of while playing with the implementation. Do we want to add/replace #uri : URI in favor of those related methods on the request interface itself. Or should we deprecate the existing methods in favor of like request.uri.path = ... instead of request.path = ...?

16

I see what you’re going for here. There’s some inflexibility in the stdlib which has made web frameworks “have to monkeypatch” in some extra nuts and bolts into the stdlib classes. Wouldn’t it be easier if we could just provide our own injected class which gets used to hold an HTTP::Response?

A solution like this is difficult because in order for it to be workable it essentially needs to be centralized. I don’t see a way forward with that. Your list of options regarding this standardization doesn’t charm me, because they all fall back to centralization.

You can make some progress with the specific Types implemented in stdlib as you’ve done here. Maybe that effort in and of itself is useful enough to framework designers that it’s worth the deprecation in the stdlib.

But what I don’t yet see is how this helps solve the general case. Wouldn’t explicit abstract-ducktyping solve these use cases as well, and more? (ie something like: abstract def(parameter : &.callable_method(arg : Type) => ReturnType)

Take the Redis example from the other thread, who would provide the standardized abstraction for what a redis client must implement? Surely the folks implementing the redis would want to do so, creating a proliferation of standards. That seems to defeat the purpose and drastically limit the effectiveness of implementing this the way you’ve laid out.

17

Centralization of what? The interfaces themselves? Or?

To be clear HTTP::Request and stuff aren’t going anywhere. All that this RFC is proposing is to like include RequestInterface into it, and the related other interfaces into their related type in the stdlib. Then it would be possible to update existing HTTP::Request parameter type restrictions to RequestInterface. This keeps things backwards compatible and everything continues to work. But it also allows others to define their own request type, include RequestInterface from the stdlib, and have the stdlib able to use it just fine.

I’m not sure I 100% follow that example. What benefits does that provide over abstract def foo(parameter : Type) : ReturnType? Where Type describes the required methods? If I’m following correctly, this just seems like a DSL to do something you can already do with a module and abstract defs.

Also fwiw, you totally can use duck typing by just not giving a type. E.g. abstract def foo(parameter : _) : ReturnType. This requires you to implement #foo, but allows any type; such that the compiler would just error if you pass it something that doesnt have all the methods defined it requires.

The problem with duck typing is that you can’t really have an ivar be set to any value. I.e. if you wanted to store the argument as an ivar to future use. This is where the interface module comes into play as it allows you to type ivar, allowing it to be anything that includes the module.

Happy to talk more on Discord or something, but the the answer I think is “it depends”. Ultimately I’m not sure there should even be a RedisInterface. Something higher level CacheInterface could deff make more sense to include in the stdlib, which each redis implementation could then implement. But a redis specific one is too specific. The stdlib doesn’t need to define interfaces for everything, because everything doesn’t need an interface.

To answer your question from the other thread (i’ll throw a link to this post in there too for posterity).

For example, using the code from the other thread:

class Library::Adapter::GaskinsRedis
  include Library:RedisInterface

  # `Redis::Client` is the entrypoint to gaskin's redis shard
  def initialize(@redis : Redis::Client); end

  def del(key)
    @redis.del key
  end

  # ...
end

The only interface you’re implementing here is your own. Because of this there should be no problems here as you’re simply mapping a concrete implementation of some specific redis client to YOUR interface. You do not need to have an interface per external lib nor have a single interface each external lib implements.

You simply expose your library’s interface it uses and users can implement it by creating a class that decorates whatever lib they want to use. Or if there is a specific one you like, you can provide a default implementation for it they can just require and use.

18

I wonder why would there be a need for an interface for http request. For example in Go that type is a struct. There’s no interface to it.

In Go you can define an interface and every type that conforms to it automatically implements the interface. That means I can create a RequestInterface and have Request implementation it. In Crystal you would do that explicitly with a module and including it.

My point is… There are some types, like http request, that don’t need an interface behind them. Some types are just data, not behavior.

4 Likes
19

My use case is that i have a custom request type for use within Athena that decorates the built in request type and is 100% compatible with it, but exposes some extra framework specific features. The interface would allow using it anywhere there is an explicit HTTP::Request type hint.

Another use case is related to HTTP::Client (which there could also be a dedicated interface for). This feature would allow custom client shards to leverage the interface, essentially allowing any client, or any shard that uses HTTP messaging, to interchangeably use request/response types from any other shard that implements it.

I.e. there would be a common interface that shards like https://github.com/icyleaf/halite/blob/master/src/halite/request.cr and https://github.com/mamantoha/crest/blob/master/src/crest/request.cr could leverage. Then you could easy use those request objects with the client defined by the shard, the stdlib client, or even like OAuth2::AccessToken - Crystal 1.11.0-dev.

Related to the previous example, if you’re making your own HTTP related lib. If you base it off the interface the end user could use whatever request/client shard they want and everything would just work. This would avoid needing to define many adapters, or tightly coupling the code to a specific implementation.

EDIT: I also found PSR-7 Meta Document - PHP-FIG which has some points, like why they went immutable. Might be a good reference, tho not all things are applicable of course.

20

Another use case is related to HTTP::Client (which there could also be a dedicated interface for). This feature would allow custom client shards to leverage the interface, essentially allowing any client, or any shard that uses HTTP messaging, to interchangeably use request/response types from any other shard that implements it.

It has already been brought up, but I think this would also make it a lot easier to pass around something like a Http::MockClient as a dependency when testing things that consume it, as opposed to something like webmock which needs to globally patch HTTP::Client.

Is the idea just to extract some of the implicit interfaces of stdlib into their own explicit modules and use them when typing? It sounds like it could be an easier win as Crystal supports everything needed out of the gate.

In the case of WebMock, I think that providing the intended state into dependencies is generally more flexible and less awkward than mutating state or a namespace.
+1 that having interfaces around some critical namespaces is a solid idea.

1 Like