tap349

tap349

How to store lots of data in memory?

Good day!

I searched the forum for similar questions but couldn’t find exactly what I need.

I use Agent process to store shared read-only data in memory. The size of data is ~10 MB.
Every user has its own handler process (implemented as GenServer process) that uses this data to calculate some result. So every time request comes, 10 MB are copied from Agent process (since everything must be immutable I guess).

I’ve noticed that the 3rd or the 4th time data is read from Agent process in each user process, it’s not fully garbage collected so memory consumed by each user process grows in size.

And if I have 1000 users, total memory consumption might grow up to 10 MB x 1000 = 10 GB which seems unacceptable.

My question is how to store lots of read-only data in memory so that it’s efficiently shared among many long-running process? Using ETS doesn’t seem to solve the problem.

P.S. data stored in memory is a parsed YML file. It occupies 10 KB in filesystem but 10 MB when read and parsed - maybe there’s a way to optimize storing it in memory as well?

Thanks.

Most Liked

rvirding

rvirding

Creator of Erlang

ETS keeps its data entirely separate from all processes so when you access an ETS you copy the between the table and the process. HOWEVER you do not copy the whole table each time only the elements you actually access. This does mean you should not store all the data in one element but in multiple elements and you the key to select which element.

Actually there is no way to avoid copying data if you are sharing data between processes, keeping it in an ETS table or storing it somewhere “outside” the erlang/elixir system. It’s a fact of life. Just make sure you can access it in small chunks.

LostKobrakai

LostKobrakai

Depending on how often your shared data does change maybe look at this one: GitHub - discord/fastglobal: Fast no copy globals for Elixir & Erlang. · GitHub

cdegroot

cdegroot

It becomes less of a surprise if you think through the implications of doing that. Erlang does garbage collection on the process level, which is very simple in multiple regards: process memory is small, and processes are interruptable so a small pause to do a quick GC is acceptable. This keeps GC simple. Now, think of the case when you would get data by reference out of a process (an Agent is just a process) - suddenly the GC has to keep track of pointers globally in the VM and simplicity gets tossed out of the window.

(the actual details are, of course, a bit more complicated than I just said. This seems to be a decent quick overview with pointers to further reading. Erlang will ask you to open the hood and look at how the engine works a bit sooner than other systems, but the pay-off is good performing stable code and the investment isn’t that high (compared to, say, learning the Java Memory Model). Well worth it.)

Where Next?

Popular in Questions Top

_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
Kurisu
For example for a current url like http://localhost:4000/cosmetic/products?_utf8=✓&query=perfume&page=2, I would like to get: ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
JulienCorb
I am trying to implement my new.html.eex file to create new posts on my website. new.html.eex: <h1>Create Post</h1> <%= ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29603 241
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36352 110
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 48342 226
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement