Have you tried processing the request in the agent process rather than the user handler process? If the request is small but the rules are large, it would require a lot less data transfer between processes. The downside is the agent becomes a bottleneck, but it’s very easy to create a pool of them as suggested above.
dom
peerreynders
You may have your reasons for not divulging any details about your processing requirements - so it is possible that you are absolutely correct that you need a separate copy of the data for each user - and then again that may simply be a superficially convenient choice.
How many simultaneous requests do you reasonably expect to be running against the data set? How large is the result that the requestor is going to get back?
In the BEAM environment it may make more sense to separate the “request” logic from the user’s client code and instead use that logic to build a short lived process that runs the logic against it’s own copy to produce the result. While that may lead to more copying it may actually require far fewer simultaneous copies of the data in memory and process termination makes GC extremely simple.
To cut down on the copying, processes could be reused a finite number of times or indefinitely (i.e. process pools as already suggested).
Copying could also be reduced/eliminated by partitioning the data in some logical way so that it could be efficiently accessed and shared (as already mentioned via ETS and/or distributing data dependent processing among multiple processes).
Shouldn’t be a surprise:
Erlang Programming 2e: Introduction: p. xiii:
Erlang belongs to the family of functional programming languages. Functional programming forbids code with side effects. Side effects and concurrency don’t mix. In Erlang it’s OK to mutate state within an individual process but not for one process to tinker with the state of another process. Erlang has no mutexes, no synchronized methods, and none of the paraphernalia of shared memory programming.
Processes interact by one method, and one method only, by exchanging messages. Processes share no data with other processes. This is the reason why we can easily distribute Erlang programs over multicores or networks.
When we write an Erlang program, we do not implement it as a single process that does everything; we implement it as large numbers of small processes that do simple things and communicate with each other.
The same essentially applies to Elixir. Sharing is convenient but that convenience comes at a cost - it’s all about tradeoffs. Furthermore some find the utility of Agents questionable while most see them as limited, see this recent topic.
Agent is a specialization that focuses entirely on state. GenServer embodies the more general notion of a process minding it’s own state and maintaining full control over access (via messaging) and mutation of that state.
In Elixir the Task is often used for short-lived processes but GenServers will still be used for one-off processing when multiple processes have to coordinate processing in the pursuit of a common objective.
GenServer is the fundamental building block, Task and Agent are mere convenience specializations that are typically only useful under the most trivial of circumstances.
4
rvirding
Creator of Erlang
ETS keeps its data entirely separate from all processes so when you access an ETS you copy the between the table and the process. HOWEVER you do not copy the whole table each time only the elements you actually access. This does mean you should not store all the data in one element but in multiple elements and you the key to select which element.
Actually there is no way to avoid copying data if you are sharing data between processes, keeping it in an ETS table or storing it somewhere “outside” the erlang/elixir system. It’s a fact of life. Just make sure you can access it in small chunks.
8
tap349
Set of rules are loaded when applications starts - they are stored in YML because they are meant to be edited by ordinary users.
Correct me, please, if I’m wrong - by compilation you mean hard-coding this data somewhere in config file? In this case it wouldn’t work because rules should be stored in user-friendly format (YML is user-friendly enough IMO 
).
So a limited number of processes (workers) have their own copies of data while user processes just pass request payload into workers to calculate the result? Well, that sounds reasonable - I’ll need to move all calculations from user processes into workers but that should not be a problem.
The only downside I see here is that I’ll be able to process max <pool_size> requests concurrently - but it must be okay for me too.
1
tap349
AFAIK agents are used for storing state only - and putting business logic inside them would break this contract. But moving logic and data access into dedicated processes (not necessarily agents) looks promising.
1
tap349
Thanks for detailed answer.
I don’t have strict requirements. I can’t say that I need a separate copy for each user - this is just how it’s currently implemented (maybe because it’s a ‘convenient choice’ or due to lack of knowledge of how to do it efficiently).
My application is kind of microservice that receives request (each having user_id and some payload) and calculates the result using this payload and the whole set of rules (the very data).
When request for new user_id is received, a separate user handler process (GenServer) is created for this user - this process reads data from agent and calculates the result. While there are no many simultaneous requests, there might be lots of long-running user handler processes (1k-10k). Result the requestor is going to get back is not large (about 1kb).
To summarize (for myself and for future readers) I have the following options:
- spawn a short-lived process (say,
Task) from inside user handler process - this task reads data from agent, calculates the result, returns it to user process and dies
(+) when the task is terminated, it’s easy for GC to clean the copy of data read from agent inside terminated task
(-) data is still copied a lot which has its own impact on performance (copying 10-20 MB in memory is cheap but still not free)
- use a limited set (pool) of long-running processes (workers) each having their own copy of data (just like @cdegroot suggested) in their states
(+) data is copied (and duplicated in memory) N times only (where N is a pool size)
This resembles solution suggested by @dom but here another abstraction (worker) is introduced between user handler process and agent.
- partition data so that it’s accessed ‘in small chunks’ (either from agent or ETS)
(+) amount of copied data is reduced - this alleviates the problem but doesn’t remove it completely
(-) in my case that wouldn’t work since I cannot partition data - I need to access it fully in each request
- use
fastglobalpackage
(+) viable alternative if the problem can’t be solved using standard OTP tools
(-) it’s an external dependency
Taking all pros and cons into consideration, the 2nd solution looks like the best one for me now.
1
peerreynders
This seems to be the most reasonable approach (in my opinion). Ditch the agent and just initialize each worker with it’s own copy of the data. Have a look at elixir_poolboy_example.
1
dom
I was suggesting that you can prototype that approach with very little change to your code, by doing the processing directly in the agent. Then if you’re happy with the performance you can ditch the agent and replace it with a (pool of) GenServer workers. You don’t need both agent and worker.
After that you can make all your workers join a process group (with pg2 for instance) and broadcast a message to that group when the config is updated.
2
cdegroot
Compilation means transforming stuff from user-friendly to machine-executable format. Read the Yaml, convert it to Erlang/Elixir/LFE code, then compile that source, and you have executable code - which seems to be what you want.
Usually, when you find you’re writing an interpreter, you’re doing it wrong. See also: Ruby 
4
rvirding
Creator of Erlang
This has been done in some packages, mochiweb iirc. The trade off is working out if I need/want to modify this data while the system is running, if so I would have to regenerate and recompile this file at runtime. Not difficult but may not be acceptable.
3