Plugin system in Elixir

Questions / Help
,
1

Hello everyone,

I read some Elixir books and I did some small toy projects with Elixir. My mind is still too object oriented programming but I want to start creating “real code”, specifically, I’d like to create shrinerb in Elixir. It is an awesome library that I use in Ruby and I think it could be cool to learn some Elixir.

My question is about the integrated plugin system. The author explained it very well in this blog post Briefly, it consists some base classes with a few methods which do very little. Then the plugins overwrite those methods adding some functionality and calling super, this way they don’t conflict each other. The user can select the plugins he needs and only those plugins are loaded. I think it is very elegant.

I don’t find a way to translate this to Elixir. The only solution I can think it is to create a “middleware engine” for each method of each class, so the plugins can inject their code safely. Two ways to do this are creating something similar to Tesla middleware or using defoverridable. However, I actually see this an overkill task. I don’t have enough experience to determinate if they are Elixir-ish solutions.

What do you think? Are good solutions? Or am I translating OOP to a functional language and it does not fit? In this case, what would be your solution?

Thank you very much for your time :slight_smile:

2 Likes
2

If I am understanding your query correct, I think what you want to look into are Behaviours:

https://elixirschool.com/lessons/advanced/behaviours/

… also… while I 100% understand the “write something I want/need to learn …” you may also want to take a look at https://github.com/stavro/arc if you haven’t already come across it. If nothing else, it may give inspiration and/or a starting point …

Cheers, and enjoy your new journey in the land of Elixir!

2 Likes
3

Hello @aseigo, thank you for your response,

I wanted something a little bit more complex, I need more composition. This small example in Ruby maybe helps:

class UploaderBase
  def upload(io, options)
    store.upload(io, options)
  end

  def delete(name)
    store.delete(name)
  end
end

class Plugin::Backup
  def upload(io, options)
    super
    backup_store.save(io, options)
  end

   def delete(name)
     super
     backup_store.delete(name)
  end
end

class Plugin::Validation
  def upload(io, options)
    if validate?(io, options)
      super
    end
  end
end

Then the user of the library do something like this:


class MyImageUploader
  include UploaderBase

  plug :backup, store_backup: :s3
  plug :validation

  def validate?(io, options)
    io.size < 3.megas
  end 
end

So basically, I want to be able to use my parent behavior and only if need it, add more functions in plugins, etc. As far I understand, I cannot achieve this with behavior, only a simple polymorphism. The middleware I linked in my previous post may work, but do it for each function for each module seems too much.

And yes, I want to do it because I will need it in the future, but mostly to learn Elixir in a more realistic scenario.

Thank you for your links.

5

So, there is the super function that does what you want with behaviours:

https://hexdocs.pm/elixir/Kernel.SpecialForms.html#super/1

You can also use the using and import directives to extend modules in nicely generic ways:

https://yos.io/2016/04/24/mixins-in-elixir/

So you can achieve what you are looking for that way … buuuuuut …

I have to say that to me, calling “super” is generally a code smell (not that it is always “wrong”, as in “worst thing one could do”, though …). Functions are cheap in Elixir, and instead of calling a magical super version of the functions, why not call the actual function you are after? The behaviour can define upload and delete, and the Uploader module can provide a set of helper functions. This allows each implementation of the behaviour to explicitly pick which functions to use. This opens doors like being able to freely compose the set of functionality that makes sense for that specific module … and if there are common composition patterns, put those compositions in a function that can be called directly … so you can get explictness, DRY, and flexibility all in one package.

The inheritance straight-jacket of “use your inheritance hierarchy as a code path” encouraged OO is often not the most expressive or flexible approach … functions are our friends, and they can be called by name :slight_smile:

3 Likes
6

Interesting. I felt the use of super/defoverridable not very idiomatic and you feel the same. That was a doubt I had.

The key point is the user of the library. He can create the uploader he wants only adding small pieces of code packaged in plugins. Currently shrine has 43 plugins! The user can as well extend the functionality easily and it can be distributed like a gem.

In my previous example, when the user calls MyUploader.upload(file, {}), the chain is: Plugin::Validation.upload → Plugin::Backup → UploaderBase. Plugins (usually, it may have dependencies) are not aware if other plugins are used. If I use the name of the function I will lose this indirection, I cannot concatenate calls.

I understand and share that OOP can be more complicated than FP in certain scenarios. However, in this one I don’t see it like OOP paradigm, what I interpret is that the author is using the tools the Ruby language gives him to create an environment where composing is easy and cheap — composing in the sense of open-closed principle, entities should be open for extension, but closed for modification — but maybe I’m wrong :102:

With Elixir I imagine I could do something like:

defmodule Plugin.Backup do
  use Plugin.Base

  def upload(io, options, next) do
    next(io, options)
    backup_store(io, options)
  end
end

So I’m creating a middleware, but shrine has 4 classes, with 4 or 5 public methods and much more private methods. Plugins modify private methods, as well. I don’t want 100 middlewares.

I don’t want to do OOP with Elixir. I want to achieve the same functionality: the user may choose the plugins for each uploader… but I don’t care exactly how to do it.

@aseigo I really appreciate your time, thank you :slight_smile:

7

defoverridable is quite idiomatic and used quite a bit. It is the use of super that is less so. Rather than going to the implementation of the function in the re-implementation, it usually explicitly implements what it needs/wants.

“super” is like having a function called “does_something”. It is non-expressive, and if it is used by every user of that behaviour then it must also be one-size-fits-all.

The exact same thing can be achieved by providing properly named functions that get called. In the example of an overridable upload function, plugins that wish the default behavior can just call Base.upload directly, with an alias Uploader.Base in the Uploader.base’s __using__ macro so you don’t have to put that explicitly in every module that uses it.

The end result is that instead of all of these anonymous and must-fit-all-sizes (a source of horrible code IME) functions, the plugins call the appropriate / desired functions from Base specifically. Something like this:

defmodule Mixin.Base do
  @callback c1() :: any

  defmacro __using__(_) do
    quote do
      alias Mixin.Base
      @behaviour Mixin.Base

      def f1, do: IO.puts "Heya!"
      def f2, do: IO.puts "Hey-ho!"

      defoverridable [f1: 0, f2: 0]
    end
  end
end

defmodule Mixin.Specialization do
  defmacro __using__(_) do
    quote do
      def f1, do: IO.puts "This is special"
    end
  end
end

defmodule Mixin.Impl do
  use Mixin.Base
  use Mixin.Specialization

  def c1 do
    # now we compose what we want to happen here ...
    f1()
    f2()
  end
end

Which gets us:

iex(9)> Mixin.Impl.c1
This is special
Hey-ho!
:ok

Which really isn’t so far off from what you had in mind; and then those overridables could be altered by other modules that are used as seen above. If you wish to control the set of plugins with configuration directives at runtime rather than at compile time such as above, a slightly different approach would be needed for the overrides … but that would also be possible with a bit of creativity.

So I don’t think you are all that far off in your thinking, I would just caution against using super() and thinking too much in terms of inheritance / hierarchies … hth

8

I don’t think this is what you’re after - but just in case - eplugin has a totally different approach to plugins.

There is a dedicated gen_server process that is responsible compiling the plugins and managing their particulars in ETS tables.

1 Like
9

I just do plugins via gen_event, isn’t that the easiest and most built-in way?

Literally I just send an event at whatever hook points I want and let whatever is listening to them do whatever they want at that point.

gen_event has it’s share of problems that I fixed in an erlang replacement library long ago, like using monitors instead of links (gen_event predates monitors did you know? ;-)). Does Elixir have a GenEvent that uses monitors? I’ve not actually checked, I just use my Erlang library in Elixir thus far… >.>

10

GenEvent is in the process of being deprecated

Plataformatec: Replacing GenEvent by a Supervisor + GenServer

11

Oh thank goodness! It is about time!

>.>

Uh, so basically what mine already is… I wonder if I should just rewrite it in Elixir and publish it so it has an elixirified API… >.>

12

If I use the name directly, plugins cannot interact with each other. Maybe the next plugin in the “middleware chain” decides to crypt the file before upload, or decides to stop the upload. This should be agnostic for plugins. super represents this — go on with the middleware — but it can also be named next_method or yield. Middleware engines are very nice IMHO.

I see I can overwrite methods, but I don’t want the user of the library to compose the logic. I showed simplified examples, but logic can be a little bit more difficult. More plugins more code and more complex.

Definitively at compile time. I don’t see the point of doing it at runtime.

Maybe my solution will be just simplifying to the maximum the API and create middleware for each method, like decorating pattern kind of.

13

I didn’t know the library. It seems very nice to decoupling the code. However, I see lack of flow control for my use case, stop the processing for example. I could define several callbacks, before_action and after_action, where the plugins can stop the action but it feels complicated.

Same here, it is perfect to decouple the code but not to control the flow of the code.

14

Callbacks.

15

In which case I would suggest looking at how plug does this. It’s an elegant approach to these kinds of situations which sidesteps the whole functions-as-API design cascade, and there is no calling of prior or parent functions in any of the plugs.

16

Message bus solutions are indeed truly powerful; but if you need / want predictable order of execution then they are more difficult (or just (d)evolve into multiple points of serialized behavior).

Not sure they are looking for a message bus solution in this case? I would expect there needing to be a predictable order of execution, and coupling steps through unwritable contracts of “when someone somewhere eventually emits message B after message A, then I’ll do something…” isn’t really so workable.

That said, I’ve used message busses in many applications in the past where order of execution wasn’t of concern, and I 100% agree with you that they are absolutely lovely in those cases.

17

Mine had priority, they registered based on a given priority, but in general it was used by plugins, not something big.