Runtime Compilation of Dynamic Modules

jasonmonroe

I am building an application which compiles new functions based on user input. For performance reasons, I would like to compile each function separately since there will be a large quantity of functions which can change individually. I already have the application working with Module.create and :ets lookups for the modules; however, I know that the design is not ideal since the module names are dynamic. This means I am creating atoms based on user input which is obviously bad.

From what I can tell, there is no way to recompile only a piece of a module which requires me to separate out the functions. I would like to avoid polluting the atom table in this way so I have come up with a possible solution.

My current idea is to create a registry process which assigns atoms to functions which they can use for their modules. As functions are removed, the atom it was assigned would be freed for the next created function to use. This way I will limit the atoms I use to the most atoms I used at any one point in time. I know entirely how I would build this but I would love to find a more elegant way of doing it.

Any suggestions would be greatly appreciated.

6 comments

#runtime-compilation

0 2210 6

2019-02-27 19:51:14 UTC

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)

axelson

Scenic Core Team

Could you rework things to use just dynamically created functions instead of actual modules? I believe that would make it a lot easier.

Post #1

jasonmonroe

Hi Axelson,

Thanks for the suggestion. For my scenario I am looking to run these functions at a really high rate and performance/throughput is my #1 goal. The benefit of compiling in my scenario is that my running processes can actually call the function directly through a naming convention rather than looking them up using an expensive process call.

To elaborate a little bit, I would like to be able to recompile an individual function while heavy execution is happening. So I need to optimize in 2 areas: execution and compilation with an emphasis on execution. If I have to send a process message to get an in-memory function, I’m certain I will be sacrificing performance.

Post #2

axelson

Scenic Core Team

Sounds like an interesting use-case! Seems similar to discord’s fastglobal, although you’d probably can’t use it directly since it’s made for storing/accessing static data, but you could probably find inspiration there:

https://github.com/discordapp/fastglobal

Do you need to return the new results immediately or can you still run the old function while the execution is happening?

Also I’d recommend that you try multiple approaches and benchmark them (e.g. with benchee | Hex)

Post #3

jasonmonroe

This is a great suggestion, I’ll look into this tomorrow and share how it goes. I’ve used Benchee for similar tests but I haven’t checked out the fastglobal code.

To make things more complicated, I actually need to store multiple versions of the functions to provide staged changes and rollback capabilities for groups of functions. I have a version of this working already, just looking to improve/innovate on it a bit.

Post #4

jasonmonroe

Just to share my result: I looked at the fastglobal code and realized this is a small part of what I am already doing. It was fun to see I came up with the same strategy through my own testing. However, I’m assuming Discord was not using dynamic value names so it doesn’t solve that problem exactly. I know what I am trying to solve is very niche but I have some direction already on how to limit the atom usage.

Post #5