tap349

tap349

How to store lots of data in memory?

Good day!

I searched the forum for similar questions but couldn’t find exactly what I need.

I use Agent process to store shared read-only data in memory. The size of data is ~10 MB.
Every user has its own handler process (implemented as GenServer process) that uses this data to calculate some result. So every time request comes, 10 MB are copied from Agent process (since everything must be immutable I guess).

I’ve noticed that the 3rd or the 4th time data is read from Agent process in each user process, it’s not fully garbage collected so memory consumed by each user process grows in size.

And if I have 1000 users, total memory consumption might grow up to 10 MB x 1000 = 10 GB which seems unacceptable.

My question is how to store lots of read-only data in memory so that it’s efficiently shared among many long-running process? Using ETS doesn’t seem to solve the problem.

P.S. data stored in memory is a parsed YML file. It occupies 10 KB in filesystem but 10 MB when read and parsed - maybe there’s a way to optimize storing it in memory as well?

Thanks.

Most Liked

rvirding

rvirding

Creator of Erlang

ETS keeps its data entirely separate from all processes so when you access an ETS you copy the between the table and the process. HOWEVER you do not copy the whole table each time only the elements you actually access. This does mean you should not store all the data in one element but in multiple elements and you the key to select which element.

Actually there is no way to avoid copying data if you are sharing data between processes, keeping it in an ETS table or storing it somewhere “outside” the erlang/elixir system. It’s a fact of life. Just make sure you can access it in small chunks.

LostKobrakai

LostKobrakai

Depending on how often your shared data does change maybe look at this one: GitHub - discord/fastglobal: Fast no copy globals for Elixir & Erlang. · GitHub

cdegroot

cdegroot

It becomes less of a surprise if you think through the implications of doing that. Erlang does garbage collection on the process level, which is very simple in multiple regards: process memory is small, and processes are interruptable so a small pause to do a quick GC is acceptable. This keeps GC simple. Now, think of the case when you would get data by reference out of a process (an Agent is just a process) - suddenly the GC has to keep track of pointers globally in the VM and simplicity gets tossed out of the window.

(the actual details are, of course, a bit more complicated than I just said. This seems to be a decent quick overview with pointers to further reading. Erlang will ask you to open the hood and look at how the engine works a bit sooner than other systems, but the pay-off is good performing stable code and the investment isn’t that high (compared to, say, learning the Java Memory Model). Well worth it.)

Where Next?

Popular in Questions Top

jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
yawaramin
In the Dialyzer docs ( dialyzer — OTP 29.0.2 (dialyzer 6.0.1) ), there is a way to turn off a specific warning for a function: -dialyzer...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New

Other popular topics Top

johnnyicon
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement