P.S. data stored in memory is a parsed YML file. It occupies 10 KB in filesystem but 10 MB when read and parsed - maybe there’s a way to optimize storing it in memory as well?

There should be, it’s 1000 times more in memory then on disk, doesn’t sound right How did you measure this difference? If you dump it into file, is it 10kb again?

Using ETS doesn’t seem to solve the problem.

If you store it there “as is” as one entry it will not bring much, but if you put the parts of that (structured) data under different keys it will help with memory consumption.

See related discussion:

Depending on how often your shared data does change maybe look at this one: GitHub - discord/fastglobal: Fast no copy globals for Elixir & Erlang. · GitHub

I don’t understand that, because storing lots of in-memory state in ETS usually works fine. Unless of course, you still copy the data out of ETS for every request…

(also, the size does not make sense; what library are you using to parse the yaml and how does the structure look like internally e.g. when you inspect it?)

Depending on your needs it might work to have a genserver that holds the data and which you can query through an api to get the parts you need. If that’s possible for you it should be able to cut down memory usage.

There should be, it’s 1000 times more in memory then on disk, doesn’t sound right How did you measure this difference? If you dump it into file, is it 10kb again?

Sorry for misleading all of you about data size - after reading file from disk I calculate and store some additional metadata. That’s why memory grows that large )

If you store it there “as is” as one entry it will not bring much, but if you put the parts of that (structured) data under different keys it will help with memory consumption.

It would be an option but unfortunately I need all this data to process each request
This data (read from YML file) contains sets of rules all of which need to be evaluated against each request.

Depending on how often your shared data does change maybe look at this one: https://github.com/discordapp/fastglobal17

That looks like I what need exactly - but I’m still a little bit surprised that Elixir/BEAM copies all the data on each read from Agent process, that there are no optimizations in this regard since it must be a common case. And copying all the time for each user doesn’t sound scalable.

I’ll give this library a try )

(also, the size does not make sense; what library are you using to parse the yaml and how does the structure look like internally e.g. when you inspect it?)

This part of my question is no longer relevant as I wrote above - YML parser must be okay (I use a fork of yamler).

I don’t understand that, because storing lots of in-memory state in ETS usually works fine. Unless of course, you still copy the data out of ETS for every request…

Frankly speaking, I didn’t try ETS solution by myself - I judge by what I’ve read somewhere on this forum: storing in-memory state in ETS is okay but when this state is read by thousands of processes, it’s copied from ETS each time leading to the problem I stated in the first post.

Depending on your needs it might work to have a genserver that holds the data and which you can query through an api to get the parts you need. If that’s possible for you it should be able to cut down memory usage.

As far as I understand, Agent is GenServer as well. So I must be already using it )
The point is that I need not the parts but all this data at once and on each request.
So using any kind of API wouldn’t help me.

Thanks for all answers - it looks like it’s a normal behaviour in OTP: when dealing with both Agent process and ETS, data is copied into client process each time.

So possible solutions I see now:

• use fastglobal package mentioned by @LostKobrakai
• tune Erlang VM to use more agressive garbage collection strategy
• kill long-running user handler processes when there are no messages in their mailboxes (maybe this is why not all data read from Agent process is garbage collected after processing request)
• reduce the size of data (I store lots of IDs - probably using something like NatSet instead of MapSet would help)

Compile the set of rules. Clearly it’s code, not data. Problem solved.

Another way - depending on the problem - would be to have a pool of workers that each have a copy of the ruleset as state and something like poolboy to route requests; that would also get rid of the copying.

It becomes less of a surprise if you think through the implications of doing that. Erlang does garbage collection on the process level, which is very simple in multiple regards: process memory is small, and processes are interruptable so a small pause to do a quick GC is acceptable. This keeps GC simple. Now, think of the case when you would get data by reference out of a process (an Agent is just a process) - suddenly the GC has to keep track of pointers globally in the VM and simplicity gets tossed out of the window.

(the actual details are, of course, a bit more complicated than I just said. This seems to be a decent quick overview with pointers to further reading. Erlang will ask you to open the hood and look at how the engine works a bit sooner than other systems, but the pay-off is good performing stable code and the investment isn’t that high (compared to, say, learning the Java Memory Model). Well worth it.)