Architecture to allow creation of third party plugins for an E-Commerce App

[arjun289]

I am working on this particular E-Commerce application, it’s a fully open source project.

It’s an umbrella with three apps:

• A core app to handle logic and db related things.
• An api app to expose APIs for customer frontend client.
• A phoenix app for the admin interface.

The next phase of our development is to design the system in a manner that it’s very simple for people to create plugins or extensions. Also, the design to integrate the plugins should be such that the admin has the flexibility to add or remove them from the admin interface.

It is similar to how Shopify allows installation of extensions. We don’t fully understand how extension installation works for individual sellers.

e.g.
Let’s say the admin wants to install a library to provide support for marketing in the system he/she should be able to install the plugin created for the purpose.
What this plugin would be capable of?

• Track User login and send a welcome email.
• Check for abandoned carts etc.

The plugin should be created in a manner that once it is installed it should be able to track the call to session controller'slogin` action and send a ping to the third party service. This can be done by somehow injecting code which comes from the extension. e.g. making a specific api call to third-party service.

Constraints:

• Can’t include the library in mix.exs as the number of packages can be many.
• Since this is a multitenant app these installations are also seller specific, every seller can have a unique combination of plugins.
• We can not have a central

One of the approaches is to inject a piece of code in the controller action to be tracked, to send an event to another library which based on metadata that the event sent(which includes a list of plugins for that particular seller) makes a decision and makes calls to the respective third party services.

What could be a possible design solution to provide the functionality?

[svarlet]

It sounds like you are trying to implement something generic and flexible for all sort of usecases.

Why don’t you start by creating one such plugin? Then a second? Then a third?

This will give you insight such as:

• is our current architecture making it hard to create plugins?
• what’s the commonalities between the first and second plugins? with the third?

You’ll certainly get somewhere, it may not be perfect but you can always rework some bits as you go. Trying to make something lile this right in one go will either lead to analysis paralysis or something unmaintainable.

[dimitarvp]

This is indeed a very generic question, but you might want to look at an event bus technology or something like RabbitMQ / ZeroMQ – basically your framework can emit (publish) events to a bus / topic and the plugins can subscribe to them and act on the data. This will give you something much easier to maintain than trying to reinvent plugin architectures.

[notriddle]

The big alternative to using a multi-server system (which is what @dimitarvp’s idea of using an event bus would mean) is to use luerl to allow “scripting” your application the same way JavaScript allows you to script the browser and VBA allows scripting the Microsoft Office apps.

[danschultzer]

@svarlet’s answer is the way I went with extensions for Pow. I created a few extensions that I felt covered a wide surface to help make the API/structure for third party extensions.

I think you’ll have to rethink the constraint of not adding the plugins to mix.exs. The app has to be compiled, and I don’t see how you can reasonably include third party plugins without requiring them in mix.

[edwildgoose]

I think this requirement is quite common and desirable.

For example lets imagine I wanted to build an email server to replace my current usage of postfix/sendmail/etc. Or I want an XMPP server (to pick an example where we can observe a plugin architecture on a high quality app)

In all these cases it’s desirable to ship the app compiled and then have the customer be able to configure it as dynamically as possible for their use case, possibly including optional functionality that they may write/create themselves

Part 1)
The lack of mix requirement seems workable because one can load any beam files you find and I guess it would be possible to dynamically start included applications if they conform to some interface. This would allow our main app to startup dependent apps which offer some kind of very dynamic plugin functionality

An alternative would be to relax the contraint on recompiling the main app and include code in the main app to start your plugins and include their code

Part 2)
You need some kind of hook architecture to allow your main app to call your plugins. This seems to be where you get furthest away from the Elixir way to do things. (Meaning that Elixir encourages everything to be explicit and it discourages clever tricks involving side effects, eg message bus and event handler solutions)

Simplest is explicitly passing hook functions to key modules. eg a module offering http client functionality could pass a callback module which implements the relevant hook code. Disadvantage of this technique is that it doesn’t scale to running multiple modules of hooks and must be defined quite explicitly in the calling code (would be more difficult to make dynamic)

Probably next up would be a plug style architecture, so that you very explicitly include plugin functionality into obvious hook functions, however, it’s much easier to specify callback chains and create a clear way for callbacks to veto further callbacks getting executed.

Next up would be that the hooks are maintained in a run time variable, so for example think gen_event or something you build yourself from genservers, genstage, etc. Dynamically things can start to listen to your hooks or remove themselves. This now allows external code to decide to compose or decompose itself from your main app. eg some config file could decide to attach some functionality to a hook. Also now you have the ability to run up multiple instances of a module and hook it to multiple hooks, eg in our mail server we might write some plugin functionality to decide if we accept/reject some email address as valid, that same module can be hooked into the hook to check both receiver address and the sender address, etc.

Finally I guess you can totally distribute the app and even remove the requirement to start all the code from the same startpoint using some kind of event bus, eg elixir registry, gproc, elixir pub/sub, kafka, etc. This has some challenges in not starting work before all plugins are attached and configured correctly

I think probably plug and mongoose/ejabberd are some good examples of how one can create a plugin architecture.

[lud]

I was thinking of using mix archives to provide plugins.

Archives would be present in a well-known directory and loaded when the app starts. I guess there is some release configuration script to write to list all archives[]/dir/ebin path to the release.

Plugin dependencies would have to be provided as archives too, just like ElixirLS does.

Plugins configuration may be an .exs file imported in runtime.exs, though there may be a lot of warnings for unknown apps. Otherwise each archive may contain a special .exs script that will be executed and whose return value would be passed to Application.put_all_env.

Anyway, sorry for the mind dump, but knowing the features of Erlang for dynamic code loading it’s a shame that there is no simple way for distributing extensible elixir standalone applications (think a forum engine, an ecommerce app like in this topic, etc.)

The other way I see is to list all paths from a directory and read in them for the application name, and then list all {app, path: dir} from deps() in mix.exs. But that means that the application must be recompiled whenever that changes. For instance, you cannot built that in a Docker container, and hope to just have to restart the container when adding a new plugin in the directory.

Is anyone familiar with the RabbitMQ plugin system internals?

[Phillipp]

Last year I started on an idea to have plugins as external applications that just connect via websocket (phoenix channels) and then communicate with the main application. I should continue on that at some time.