Seeking Feedback on Building a Versioned Core System with Extensible Plugin Architecture in Elixir

Questions / Help
1

Hi everyone!

I’m working on an open-source project built with Elixir and Phoenix. The product (not public yet) is designed to be a modular system with a versioned core and an extensible plugin architecture. The core includes a set of plugins maintained by the product owners and a UI, but customers running their own product instance can also add custom plugins without modifying the core system. Those plugins could also extend the UI.

Here’s what I’m aiming for:

  1. Versioned Core System:

• Customers should be able to upgrade the product to a new version while ensuring backward compatibility with their existing plugins.

• I’m thinking of using semantic versioning and providing a clear API/contract for plugins.

  1. Extensible Plugin Architecture:

• The product will have both built-in (core) plugins and external/custom plugins.

• Customers should be able to write their own plugins and integrate them into the product without recompiling the core system.

• Ideally, plugins would be dynamically loaded at runtime or managed in a way that avoids the need to rebuild the release.

  1. Challenges:

• Handling upgrades in a way that preserves stability for custom plugins.

• Allowing customers to easily build and integrate their plugins, given that Elixir releases bundle and compile the system.

I’m particularly inspired by systems like Kubernetes’ extensibility and Livebook’s Smart Cells, which allow dynamic additions and version compatibility. I’d love to hear from the community:

• Have you worked on similar systems in Elixir or other languages?

• What approaches have you used for plugin architecture and version management?

• Any tips for handling dynamic loading, plugin API contracts, or upgrade paths in Elixir?

• Are there libraries, patterns, or tools in the Elixir ecosystem that you’d recommend?

I’m looking for guidance, feedback, or even cautionary tales as I continue building the product. Thank you in advance for your insights!

5 Likes
2

I’ve built a few systems having templates+code defined by users, there are few core concepts to keep in mind:

  1. Erlang VM has no concept of sandboxing, running code at runtime has the same security implications as your application code. Dune tries to address this, but be sure to read the security guarantees: GitHub - functional-rewire/dune at v0.3.10
  2. There is luerl to address the sandbox issue, but you obviously have to write lua, which changes the way you do things;

As for everything else like versioning, that should be the easy part, just work with callbacks that provide data as arguments and don’t allow side-effects, if you have to do side-effects under the hood, build a declarable graph as a response from client code that you can process later, that will also help should you want to change something down the line.

3 Likes
3

Hey this sounds like exactly the problem that the WebAssembly component model evolved to solve. I am hard at work adding support for it to the Wasmex library. In a nutshell it will let you define a high-level typed interface that you can implement in any language that targets the component model (Rust, JS, Python, C#, etc) and be able to execute the component in a secure sandbox. I think it will be the future of software extension. I need to get my PR mergeable and then write an example blog post of how to use it in Elixir.

4 Likes
4

Are you using WIT for that?

5

Yes, that’s part of the wasm component spec.