I think this requirement is quite common and desirable.

For example lets imagine I wanted to build an email server to replace my current usage of postfix/sendmail/etc. Or I want an XMPP server (to pick an example where we can observe a plugin architecture on a high quality app)

In all these cases it’s desirable to ship the app compiled and then have the customer be able to configure it as dynamically as possible for their use case, possibly including optional functionality that they may write/create themselves

Part 1)
The lack of mix requirement seems workable because one can load any beam files you find and I guess it would be possible to dynamically start included applications if they conform to some interface. This would allow our main app to startup dependent apps which offer some kind of very dynamic plugin functionality

An alternative would be to relax the contraint on recompiling the main app and include code in the main app to start your plugins and include their code

Part 2)
You need some kind of hook architecture to allow your main app to call your plugins. This seems to be where you get furthest away from the Elixir way to do things. (Meaning that Elixir encourages everything to be explicit and it discourages clever tricks involving side effects, eg message bus and event handler solutions)

Simplest is explicitly passing hook functions to key modules. eg a module offering http client functionality could pass a callback module which implements the relevant hook code. Disadvantage of this technique is that it doesn’t scale to running multiple modules of hooks and must be defined quite explicitly in the calling code (would be more difficult to make dynamic)

Probably next up would be a plug style architecture, so that you very explicitly include plugin functionality into obvious hook functions, however, it’s much easier to specify callback chains and create a clear way for callbacks to veto further callbacks getting executed.

Next up would be that the hooks are maintained in a run time variable, so for example think gen_event or something you build yourself from genservers, genstage, etc. Dynamically things can start to listen to your hooks or remove themselves. This now allows external code to decide to compose or decompose itself from your main app. eg some config file could decide to attach some functionality to a hook. Also now you have the ability to run up multiple instances of a module and hook it to multiple hooks, eg in our mail server we might write some plugin functionality to decide if we accept/reject some email address as valid, that same module can be hooked into the hook to check both receiver address and the sender address, etc.

Finally I guess you can totally distribute the app and even remove the requirement to start all the code from the same startpoint using some kind of event bus, eg elixir registry, gproc, elixir pub/sub, kafka, etc. This has some challenges in not starting work before all plugins are attached and configured correctly

I think probably plug and mongoose/ejabberd are some good examples of how one can create a plugin architecture.

It sounds like you are trying to implement something generic and flexible for all sort of usecases.

Why don’t you start by creating one such plugin? Then a second? Then a third?

This will give you insight such as:

• is our current architecture making it hard to create plugins?
• what’s the commonalities between the first and second plugins? with the third?

You’ll certainly get somewhere, it may not be perfect but you can always rework some bits as you go. Trying to make something lile this right in one go will either lead to analysis paralysis or something unmaintainable.

The big alternative to using a multi-server system (which is what @dimitarvp’s idea of using an event bus would mean) is to use luerl to allow “scripting” your application the same way JavaScript allows you to script the browser and VBA allows scripting the Microsoft Office apps.