Create a new module at runtime based on existing code and save it in database

Questions / Help
1

So I have the following use case.
I have some code / modules that I want to run different versions of.
Let’s say I have a user, when they sign up - they interact with some code v0.
I now want to deploy new changes to the that code, but this first user should still interact with the code of v0. Another user with v1 and maybe some other on v2.
At some point in time the user could decide to update to v1 or v2, but it’s their decision.

I was thinking, if I could do something like:

Module.create(ModuleVersionX, module_contents, ...)

bytecode = capture_bytecode_of(ModuleVersionX)
save_bytecode_to database

// later
bytecode = capture_bytecode_from_db()
reload_module(bytecode)

My specific questions:

  1. how do I specify module_contents based on an existing compiled module?
  2. I think I can use @after_compile to capture the bytecode of this new module? If I’m able to add this @after_compile hook in the module_contents
  3. I’m not sure how to load the bytecode again from the database.
  4. What are the catches of an approach like this?

I found the following topic that might be interesting, but the link in there doesn’t work anymore: Any ideas on how to generate compile-time module based on other modules?

2

What’s the issue in simply keeping version field for each user and just have all versions of the module in your codebase? And then just use user.code_version in a case statement and dispatch to the right module + function?

3

I’m not actually keeping track of versions manually. Most (but not all) of the code of the module is just text/markdown and changes often.
And multiple people can make changes to it.

So that would mean I need to keep track of all versions manually. I don’t want to do that.

In the future I might take a similar approach of what I think beacon is doing (compiling the module from text saved in the database), but right now that would be a big change that I don’t want to do.

4

I see. Indeed in this case it’s best to just save the compiled bytecode to the DB.

5

Yes, so my question is how to do that. Well not just saving, but then using it again etc…

6

I have only minimal experience with this but I think you’d use a function like :code.get_object_code/1 to get the binary representation of a module :code.load_binary/3. Although if you’re running on IEx then you could get the binary code directly from the defmodule result: {:module, _mode, binary, _} = defmodule Bench do def run do 42 end end and use Code.eval_string to start from a raw string:

{{:module, _mode, binary, _}, _} = Code.eval_string("defmodule Bench do def run do 42 end end")

Also I’d recommend storing the source code representation alongside the compiled binary. That will likely be necessary for checking for abuse, and also if you upgrade your Elixir/Erlang version.

1 Like
7

You need to enforce unique names and figure out how to make the correct one be called. This is also effectively a remote code execution attack vector, so you want to take good care of who is able to affect those modules.

If there’s no hard need for elixir you can consider luerl / lua (Lua — Lua v0.0.20) instead, which would allow you to retain the ability to code up logical parts, while being able to be fully sandboxed.

If this happens to be around templating then I’d suggest GitHub - edgurgel/solid: Liquid template engine in Elixir, which would be even more focused.

3 Likes
8

If I’m not mistaken I think BeaconCMS does something similar?

9

yes, but they start from a string/code representation that they compile, and they overwrite the existing module.

I want to create a new module each time - and preferably not from code but from the bytestring

That’s not really an issue:

  • enforce unique names: hash the bytecode.
  • call correct one - save the hash for the user - the hash is the version number
  • remote code exectution: only devs with access to the repository make changes - so they can already do whatever they want.

I don’t need a sandbox. I just need a way to version code at runtime.

The flow would look something like this:

# in application startup

hash = byte_code(module) |> some_hash_function

previous_version_hash = get_previous_version_hash()

if hash != previous_version_hash do
  # create new module with name: `Module-#{hash}`
  save_new_module(name, bytecode)
end

# for new users

module_name = fetch_current_version()
save_module_name_for_user(module_name, user)

# for existing users

module_name = find_correct_version(user)
bytecode = select bytecode from * where hash = hash_for_user;
load(bytecode)

I’m not looking for a perfect solution. Just something that works good enough.

10

:code.load_binary seems to be a big part of the solution. Thanks

:code.get_object_code doesn’t seem to work for dynamically defined modules - at least in iex. It does work for

iex(1)> defmodule Foo do
...(1)> end
{:module, Foo,
 <<70, 79, 82, 49, 0, 0, 4, 84, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 161,
   0, 0, 0, 16, 10, 69, 108, 105, 120, 105, 114, 46, 70, 111, 111, 8, 95, 95,
   105, 110, 102, 111, 95, 95, 10, 97, 116, ...>>, nil}
iex(2)> :code.get_object_code(Foo)
:error
iex(6)> :code.get_object_code(Elixir.Foo)
:error

But that doesn’t matter that much, because I get bytecode back from Module.create.

I guess to use Module.create though, I’ll need to have the actual code available?

11

I think I found something that can actually help me fix this.

It’s not exactly what I’m looking for but it gets me a lot closer I believe: The horus application

From the readme:

Horus is a library that extracts an anonymous function’s code as well as the code of the all the functions it calls, and creates a standalone version of it in a new module at runtime.

The goal is to have a storable and transferable function which does not depend on the availability of the modules that defined it or were called.

Found it here: Horus - extract an anonymous function as a standalone module - #7 by dumbbell - Libraries - Erlang Programming Language Forum - Erlang Forums

4 Likes
12

I really hope this is a personal project and not something some other poor developer will inherit down the line.