Noob Question: Is it possible to enable user contributed 'extensions' to your compiled elixir application?

[null0perat0r]

A little background here. I have a java web application that i’m considering porting over to elixir. One key feature my java app has is the ability for admins to extend or replace certain functionality by building their own custom jar files. All the admins need to do is drop in their extension jar file in a certain directory and the web application would handle the rest.

One example of this use is the web application has a search feature that uses its internal database by default. Admins could drop in their custom jar file, restart the web app, and now the the search feature can use their own API to return the results they want. They never need to know build the main project or have any knowledge of the web app source code.

Is there anything similar that can be done in elixir where the main project can check for the existence of new libraries in an external folder and if so execute this functionality without having to rebuild the main project.

[kokolegorille]

Hello and welcome,

I don’t think it’s possible to do something similar.

One main reason would be, it’s not secure to let people add modules on the BEAM.

It’s possible to load modules, or even change their code while the system is running… but not like You describe.

[kip]

You might consider using lua via luerl as an extension language since it is properly sandboxed and designed as an extension language.

[v0idpwn]

Yes, you can. But this doesn’t mean it is well supported or safe. You can even load new modules in runtime (without restarts!).

[null0perat0r]

Many of you said this something related to safety. How would letting admins add their own modules be any less safe then having admins rebuild the entire project? I feel like i’m missing a core elixir concept here

[kip]

That’s a fair callout. BEAM operators typically have an expectation of very long running apps that just ‘stay up’. One difference between the two scenarios is that loading something arbitrary into a live system carries more risk than rebuilding (and the assumed re-testing).

That said, you can certainly load beam files into a running system. You would need to do something like:

1. Have a process scanning for new files
2. Attempt to load the file with Code.load_file/1
3. Call some well know callback in the loaded module to link things up

[axelson]

@kip

That said, you can certainly load beam files into a running system. You would need to do something like:

1. Have a process scanning for new files
2. Attempt to load the file with Code.load_file/1
3. Call some well know callback in the loaded module to link things up

If you want an example of that you might want to look at GitHub - falood/exsync: Yet another elixir reloader.
Specifically the BeamMonitor module. Although exsync itself isn’t meant to be used in production.

[hauleth]

You do not even need that. All you need is to be able to add path to the paths list (via Code.append_path/1) and then just use the module you want. Of course if you need to, then you can traverse that directory on demand and then just call these callbacks, but I would not do such thing automatically, I would have list of “enabled” extensions, and call only them, the rest would be treated only as “available to use”. Thanks to that you can also avoid problems with the dependencies, as such dependencies would be automatically loaded if present in “extensions directory”.

[kip]

Ohhhh, I though that autoload was only true if not in production but it seems I was mistaken (not the first time!).

[hauleth]

AFAIK you cannot disable autoloading, what Erlang does in production is preloading all modules beforehand to improve initial response time, it doesn’t mean that new modules cannot be loaded at runtime in addition to existing ones.

[bgrommes]

Is there a type discovery system in Elixir? What I’m accustomed to is being able to access an assembly, discover the types it contains, instance a type, call methods or read/write to properties, etc. in those infrequent situations where I need runtime binding rather than early / compile-time binding. For those familiar with it, I’m referring to reflection.

The use case I recently encountered was getting around a vendor’s poor design for customizability, that forced a certain way of organizing code which they compiled at runtime. Over the years, one source file had grown to over 42,000 lines of code.

To reorganize that without breaking the existing extensibility mechanism, and meet the client objective that test support code had to be in its own .dll that would not be loaded, run, OR distributed to the production environment, reflection proved indispensable.

More typically, I wouldn’t be doing runtime code loading or type discovery, but things like loading a type based on its name so that you could make certain logic configurable, e.g., “let’s plug in the logic for customer X here based on something set in a dashboard UI that isn’t known at compile time”.

I realize that in the functional world we don’t have user-defined types (classes) in the same sense, but I wonder what the equivalent would be in the Elixir world.

[kip]

Some thoughts for consideration, but not a proposed architecture.

As you recognise, the functional model and the ‘object’ model organize data and code differently. Both can encapsulate code and data, both can represent forms of polymorphism, object orientation favours inheritance whereas the functional model favours composition.

Introspection

In Elixir (erlang, BEAM, OTP) the fundamentals exist:

• For modules, module.module_info/0 (and module.module_info/1) can be used to identify the functions that are exports and much more. For example:

iex> Calendar.module_info
[
  module: Calendar,
  exports: [
    __info__: 1,
    get_time_zone_database: 0,
    put_time_zone_database: 1,
    strftime: 2,
    strftime: 3,
    truncate: 2,
    behaviour_info: 1,
    module_info: 0,
    module_info: 1,
    compatible_calendars?: 2
  ],
  attributes: [vsn: [64017337862377723512439896651098344616]],
  compile: [
    version: '8.0',
    options: [:no_spawn_compiler_process, :from_core, :no_core_prepare,
     :no_auto_import],
    source: '/home/build/elixir/lib/elixir/lib/calendar.ex'
  ],
  md5: <<48, 41, 74, 113, 137, 7, 188, 147, 27, 101, 152, 222, 121, 121, 204,
    168>>
]

• You can also leverage the info/1 function that is added to each module at compile time

Encapsulation

Elixir provides behaviours and protocols to support a form of polymorphism. Behaviour is a way to enforce a consistent interface to a module and therefore can be used to define different concrete implementations for an abstract concept. In Elixir, for example, the Calendar behaviour defines the functions required to be implemented in any module that supports the Calendar behaviour. You can see if a module implements a behaviour:

iex> Calendar.ISO.module_info(:attributes)
[vsn: [261504321433274647483801775536816486412], behaviour: [Calendar]]

You can also find the callbacks defined for a behaviour with the following although this is probably of limited use since if you know the behaviour then you know the callbacks. And since you can’t really introspect the parameters or identify the semantics of the functions there’s not really anywhere to go.

iex> Calendar.behaviour_info :callbacks
[
  year_of_era: 1,
  valid_time?: 4,
  valid_date?: 3,
  time_to_string: 4,
  ...
]

Protocols are commonly defined to present a standard functional interface to data types (including, and very commonly, structs). The Inspect protocol is probably the most well known but its easy to implement your own protocol and implement it for the types you require.

Application to extensions

In considering extensions, behaviours and protocols would likely form the basis of the architecture. Introspection of a given module can be used to determine if it implements a given behaviour.

• Module loading is, as @hauleth noted, a matter of having the BEAM file in the code load path. To be more deliberate you can use Code.load_file/1.

• System-wide discovery of which modules implement a given behaviour is less straight forward and this article and the accompanying library behaviour-relfection may give you additional ideas.

With the tools above you could try a naive extension approach by:

• Load a user-supplied BEAM file and capture its module name from the file name
• Check if the module implements the expected extension behaviour
• Register the module into your core application callback mechanism (this part up to you)

[qhwa]

Maybe you don’t need to do the same thing as your Java application. I think your application needs jar files only because that’s required by JVM.

In Elixir, modules can be defined at run time so the admin can just upload valid .exs files and you can run them on the backend. As a side effect, the .exs files can define some modules.

I’ve created a small library called formular and in the latest development, I’m doing similar things. Compiling to Elixir modules has been supported by the master branch and version 0.3.0-pre.2. It may not fully fit your need because not everything is allowed to run by default, but I think that would be easy to change, and most importantly, it’s a demo of compiling modules on the fly.

The idea is that source files are easier to distribute and verify (since the awesome APIs on dealing with ASTs) than BEAM files.

[bgrommes]

Thanks for this excellent overview. it’s very helpful – to me, anyway – to relate bits of functionality in the stack I’m familiar with, to how one would do it in a different stack.

While the OOP world has tended to fixate on inheritance, the original basic insight was that of message-passing, and I think that the better and more seasoned designers in the OOP world focus on composition over inheritance and consider deeply-nested inheritance trees to be code smell. Personally, I think about system composition first and foremost, and any inheritance I do use is quite “shallow”. But of course you are still left with mutable instances as a basic metaphor, which is what functional tries to get away from, in the interest of safer multi-threading. It is definitely a paradigm shift thinking in terms of function pipelines rather than organizing around types.

[kip]

Very fair callout. I am guilty of writing some of the worst Ruby code ever - monkey patching classes in the pursuit of cleverness, dynamically fiddling with the inheritance chain and then never being able to work out what the heck was going on!