Abstracting out adapters - Dependency Injection

One of the things I appreciate about the design of Shards is the ability for multiple owners to contribute their own shard for a purpose – I can build a redis adapter and share it, and so can you. And someone else. We can work together and avoid a landrush.

But how do I write a library that is capable of handling “whatever redis” you give me?

It’s easy for me to write this because I know the type of redis_url – a string:

require "redis"

module Library
  class Config
    property redis_url : String
  end
end

Library::Config.new.redis_url = "redis://localhost:6379"

It’s also easy for me to declare that I might want property redis_instance : Redis::PooledClient if I know that you’re using a Redis library that provides that feature.


What I can’t find a solution for is as follows. I’d like for the user of my library to be able to pass in an instance of an unknown class, which obeys an interface:

module Library
  class Config
    property redis_instance : ???
  end
end

Library::Config.new.redis_instance = Redis::Connection.new "redis://localhost:6379"

In this case, I only need the redis instances to have these methods: #hget, #hgetall, #hset, #expire, #del, #ttl, #increment, #hincrby, #flushall, #lpush, #lrem, #zcount.

  • Typing redis_instance as a generic Class doesn’t work because the compiler doesn’t believe that the methods I need will be defined.
  • Inheritance doesn’t work – this would create an N-interfaces for N-projects problem
  • Duck Typing would work but I don’t know how to lead the compiler, if it’s even possible.

Library flexibility isn’t the only reason I’d like to achieve this. I’d also like to share the connection pool of the host application. For other databases, this could be an important and even expensive distinction – if embedding my library in an app requires 2 connection pools, that may require someone to pay for a higher tier of a database or be faced with limited horizontal scaling because the number of connections is too high.

So basically the solution to this is Dependency inversion principle - Wikipedia. This use case is the core reason Athena provides a bunch of interface modules. It allows Athena to base its implementation on the interface itself, and not a specific concrete implementation. I.e. If you wanted to customize some specific part of it, you can define a custom implementation, include the module, and tell it to use your version and everything continues to work. Assuming you implemented the interface correctly.

Another good example are the DB drivers in core Crystal org. I.e. there is crystal-db that defines the “interfaces” for the various common parts of DB system. Like the connection, the driver, the query methods etc. Then each specific DB shard, requires this lib as a dependency, extending/including its interfaces to provide driver specific implementations.

Because each specific DB uses the same interfaces, they are all interchangeable (at least in regards to the abstractions setup within crystal-db). Some DBs have unique features that would be ofc limited to that specific implementation.

For the most part in Crystal, there is no dedicated “interface” type. The best option that works essentially just as good, is a module with abstract defs. E.g.

module RedisInterface
  abstract def hget
  abstract def hset
  
  # ...
end

Another option would be to go a step further, and maybe don’t have an interface for redis itself, but for a higher level feature like CacheInterface that would allow you to use redis, or memcached, or even a DB or the file system. I’d also checkout [RFC] Standardized Abstractions, as that is focused on the same sort of problem, and shows some examples of how PHP handles it. Doing something similar for Crystal may also be beneficial.

EDIT: I missed the core question. The solution to “handle any redis” is to ultimately have just a single robust redis shard everyone uses, versus some people using shard A and other B, and even others C. A workaround for this would be to have your lib depend on RedisInterface that it defines, then create adapters for each redis shard that implement that interface. Then the user would have to require whatever adapter they’re using (or maybe you could detect it somehow via a macro and require it for them).

@crimson-knight Went through a similar exercise, so he can prob provide some more specific insight as well.

I was just posting a lengthy reply about what I wish we could do… But yes, the exercise just showed me I still have much to learn sensei :sweat_smile:

Haha just a little humor, you’ve got the patience of a saint my friend!

So, you’re right on the money that the community would need a single “redis” shard that did everything we need. But, that would mean we need a tool like Ruby Gems but for Crystal (which I’d love, but that’s a bigger hurdle to overcome I think).

I think it would be worth adding decoupling behavior to the Crystal language though. Right now it’s basically impossible to write a shard that accepts any library that would meet any interoperability requirements.

You’ve probably felt this pain at some point though, being a framework maintainer in Crystal and being required to know exactly what all your dependencies are means you have a very tightly coupled system. It would make a much more loosely coupled shard environment if we could create some kind of failsafe syntax. I have this wild idea that we could have a class definition with the required methods and some minimal default behavior in the event the required class isn’t passed in.

module Library
  class MyClass
    property redis_client : Redis

    def initialize(@redis_client)
    end

   failsafe class Redis
     # These methods are checked against any @redis_client to ensure they respond_to? the necessary methods
     required_methods: :hget, :hset

    failsafe def initialize
      # Initialize the class in the event nothing has been passed in for the @redis_client
    end

    # When the client is initialized, use this default behavior
    failsafe def hget
      # put your desired default/failsafe behavior here
    end

    failsafe def hset
      # You've got this figured out by now
    end
  end
end

The idea here is that either A: the expected redis_client with the type Redis is passed in and is checked for the required methods to be present (arity would already get checked by the compiler, right?) or B: the Library::MyClass is initialized without the client being passed in, and the compiler would use the failsafe class definition in it’s place as if it were passed in (skipping the required_methods check, ideally).

Something like this would allow shard maintainers to create a public interface that allows for drop-in replacement shards. I really think if we aren’t going to have something equivalent to a Ruby Gems for Crystal, we should have something that allows decoupling our libraries. It doesn’t have to be this exact syntax, but something enabling this type of behavior would be AMAZING.

I’ll have to read the wikipedia article you linked sitting next to my compsci bookshelf :wink:

I think this is what you mean, translated to this specific example:

##### in my library
module Library
  module RedisInterface
    abstract def del(key)
    abstract def expire(key, seconds)
    abstract def flushall
    abstract def hget(key, field)
    abstract def hgetall(key)
    abstract def hincrby(key, field, increment)
    abstract def hset(key, field, value)
    abstract def keys(pattern)
    abstract def llen(key)
    abstract def lpush(key, value)
    abstract def lrem(key, count, value)
    abstract def rpoplpush(source, destination)
    abstract def ttl(key)
    abstract def zadd(key, score, value)
    abstract def zcount(key, min, max)
    abstract def zrangebyscore(key, min, max)
    abstract def zrem(key, value)
  end

  class Configuration
    property redis_connection : Library::RedisInterface?

    def redis_connection! : Library::RedisInterface
      redis_connection.not_nil!
    end
  end
end

##### in an implementation of my library
require "redis"
require "library/redis_interface"

# Monkeypatch in my named type, so Library can match on it and the compiler
# knows that all of the methods I'm going to use are defined.
class Redis
  include Library::RedisInterface
end

Do I have that right? If so, that’s at least a path forward. It feels like there may be some gotchas there. I’m unsure how far down the road of typing I can get in that interface. I’m left wondering what happens with named method parameters, too.

Is that not just shards? Granted I’m not super familiar with Ruby, but my assumption was it’s just their package manager for libs.

Not impossible, but deff requires extra boilerplate to properly handle. I.e. all the adapter logic. It also wouldn’t be out of the question to require the user to implement their own adapter/type based on some interface. This would remove the burden from you, but would ofc make it a touch harder to use. This could be a viable solution for cases where there are too many possible libraries you want to integrate with, or if they want to use a lib that you don’t provide a built in solution for.

Is this not the same thing as your library providing a default implementation of an interface also defined by your library? E.g.

def initialize(redis_client : Redis? = nil)
  @redis_client = redis_client || DefaultRedisClient.new
end

My main point of friction is [RFC] Standardized Abstractions, which could be solved right now by adding more interface modules to the stdlib, or even having a dedicated shard to contain them, and using them in certain places. This would not require any new compiler/language features.

As I mentioned in that other thread, PHP handles this via PHP Standards Recommendations - PHP-FIG , which define interfaces/requirements for various common libraries/implementations, such as eventing, caching, and HTTP clients, factories, and messages. They also have it somewhat easier for Redis in particular since there is only 1 implementation, the C extension.

Depending on the use case, it may not actually be a problem if we do something similar to PSR in that there is a more general CacheInterface that libraries could depend on, with each Redis implementation, implementing that interface. Granted this wouldn’t solve the issue of easily switching Redis implementations, but it might if you’re just using it power some more generic caching layer.

Close, but instead of monkey patching the external redis library yourself; you would (or require the user to) provide additional types(s) that wrap the external lib implementation to transform its methods to your interface.

Something like:

class Library::Adapter::RedisA
  include Library:RedisInterface

  def initialize(@redis : External::Redis::A); end

  # Define methods that map the API of the external shard to your API
  # Could also maybe leverage `delegate` if they're the same
  def del(key)
    @redis.delete key
  end

  # ...
end

As you pointed out, the main benefit of this is you are totally decoupled from both what underlying redis client you’re using, and insulated from changes within that lib. I.e. you can be as type strict as much as you want, as long as you do the required transformations to make each external redis shard adhere to your interface. E.g. using .as as needed or what have you.

The point in all of this is making it so your code only depends on the abstractions/interfaces it defines. Then it become super easy to plug in implementations other than redis if you really wanted as nothing says, unless you want it to be, that your Library::Config type depends on redis, but maybe just ConfigInterface or CacheInterface as i mentioned earlier.

The only gotchas with this approach is 1) the extra boilerplate required in your (or the user’s code), and 2) dealing with breaking changes in the underlying library that affect its public API and by extension your adapter. Tho, in this case you could either do version comparisons with macros, or just define another adapter for that version :person_shrugging:.

Also checkout [RFC] Standardized Abstractions, as it’s also related to decorating types.

Just wanted to add that you can declare types on those abstract methods for the interface:

module RedisInstance
  abstract def flushall : Nil
  abstract def zcount(key : String, min : Int32, max : Int32) : Int32
  abstract def hgetall(key) : Array(String)
  # etc.
end
 
class MyClass
  include RedisInstance
 
  def flushall : Nil
    puts "flushed"
  end
 
  def zcount(key : String, min : Int32, max : Int32) : Int32
    1
  end
 
  def hgetall(key) : Array(String)
    [] of String
  end
  
  # etc.
end
 
my_class = MyClass.new
if my_class.is_a?(RedisInstance)
  puts "my_class is a RedisInstance"
else
  puts "my_class is not a RedisInstance"
end
# => my_class is a RedisInstance

Also, I second that RFC.

1 Like

Yeah, that’s what I intended to show happening. The “install instructions” would highlight needing to decorate whatever Redis class with my abstract module as I have done in my example. If there were some more lifting necessary, where some adapter methods were required they can be included there too. For Redis, this is something that’s not really an issue because everything is strings. I think we’re tracking on the same page.

Re Rubygems, I think @crimson-knight is talking about RubyGems the website and central distribution mechanism for gems, but they’re also the folks behind the cli gem tool – gems can be installed from git, but the default is to pull them from rubygems.org. The point being that if there’s a central source then de-facto standards become more obvious. This has been rehashed a dozen times over and over, and over, and over.

Perfect yea. Could also provide a default adapter for your library’s preferred Redis lib. That would make getting started a bit easier, while also leaving the extra flexibility.

Ahh okay, that makes more sense. Yea I don’t think shards is ever going to go centralized, but I think we have some options to make that easier.

It is, but it’s a (mostly) centralized one. If you define a single source for Ruby gems in your project (this is the most common usage), you’ll never have naming collisions like we do with Crystal shards.

In Crystal 1.5, Warn on positional parameter mismatches for abstract def implementations by HertzDevil · Pull Request #11915 · crystal-lang/crystal · GitHub “Warn on positional parameter mismatches for abstract def implementations” means this technique will probably be difficult to implement as I’ve represented it here because not only the method names but also the parameter names will have to be in sync across various implementing libraries. I find that unlikely even in the most rigorous specifications.

I don’t think it’s a deal-breaker though. The same pattern can be used this way:

module Library::GaskinsRedisInterface
  abstract def expire(key, seconds)

  # code from Library will call this method:
  # - interactions of types can be cleanly isolated here
  # - eventually this will delegate to the abstract def above
  def library_expire(key, seconds)
    expire key, seconds
  end
end

It’s laborious at best, but it’s workable. It provides a place for an adapter pattern if the interfaces are disparate enough as well.

1 Like

Answered in [RFC] Standardized Abstractions - #17 by Blacksmoke16. Posting here for posterity.

1 Like