james-bowers

james-bowers

How do I limit the size of an ETS table to avoid a memory crash!?

:wave: hello!

Context
We are planning on using ETS to cache web responses for up to 6 hours.

Question
Is there a neat way of limiting the number of records stored in an ETS table? I’m worried that if we do not have a way of limiting this, our instances will crash if we suddenly have a burst of variations that need to be cached.

Possible solutions ?

  • Store the time when each record was inserted with the cached content, and have a GenServer periodically query and delete the oldest records. However, this could still crash if the table isn’t “maintained” frequently enough.

  • Upon each insert, we could query the size or number of items in the ETS table, but I imagine this would be slow unless an ets table itself stores the count of items?

  • Maintain a second ETS table that keeps count of the cached items. However, I’m unsure of how to keep that in sync with the cache table when records expire.

  • Use DETS, as there is far more disk than memory space, but that just postpones the issue.

At some point, we will consider using mnesia to share these cached responses with the other instances we have running.

Any feedback on those options, or an alternative solution entirely would be much appreciated!

Marked As Solved

Fl4m3Ph03n1x

Fl4m3Ph03n1x

Welcome to the forum! You pose a interesting question where many solutions are viable. You already thought about some, so lets dissect them a little.

You could have a GenServer check your ETS table every minute. Afraid this GenServer that clears the table may die? No worries, have Supervisor for it! Simplicity at its finest.

Worried one minute may be too much? You can use select_count and use an inverse exponential backoff algorithm for the pooling time (the bigger the table, the quicker you check).

You can use the select_count I mentioned before and it will be reasonably fast, while still being atomic. you can have each operation do an insert and then and update_counter but then you won’t have atomicity.

If using select_count is too slow, imagine having to write in two different tables! No way it’s gonna be faster and you still have to sync them! Is sync really important to you? Or is it OK to be a few values away from reality?

I don’t recommend DETS overall. Last year we had to remove DETS from all our systems because we were such under heavy load that our DETS tables were saving corrupt data. They just couldn’t keep up. We moved to ETS and we never had a problem again.

Furthermore, caches are supposed to be fast, and IO access to disk is by far one of the slowest things you can ask your machine to do, together with network requests. So I wouldn’t advise it.


@outlog cachex looks really cool!


Alternately a few weeks ago I posted a similar issue. I don’t check for ETS size, instead I check for the machines available RAM using memsup. You can see the original topic here:

Hope it helps!

Also Liked

outlog

outlog

look at (and use?) cachex: GitHub - whitfin/cachex: A powerful caching library for Elixir with support for transactions, fallbacks and expirations · GitHub

limits are here https://github.com/whitfin/cachex/blob/master/docs/features/cache-limits.md
though currently it only supports limiting amount of keys - not memory usage.. but limiting keys could be good enough..

Fl4m3Ph03n1x

Fl4m3Ph03n1x

I didn’t bencharm select_count VS :ets.info so I can’t give you guarantees. However, from the documentation I understand that :ets.info is O(1) while select_count is O(n) because it needs to check every element for the condition you provide.

Therefore, based on this premise, I would recommend :ets.info instead of select_count.

If you have the chance you can confirm it via benchmarks (you can use benchee for that) and then post the results. It would be interesting, at least to me :stuck_out_tongue:

tty

tty

There isn’t any options in ets which would limit the number of records in a table. ets is bounded by the amount of memory you have on the server. How large do you expected ets to grow and how much memory do you have ? We have routinely gone into GBs ets table size without issues.

If your records expire based on time you might want a round-robin ets data structure to make it easier to drop an entire set of records.

mnesia is based on ets and if you have concerns with the size of your tables exceeding your memory moving to mnesia would not resolve this concern.

Where Next?

Popular in Questions Top

siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
mgjohns61585
Could someone help me? I’m making my first elixir program, number guessing game. I can’t figure out how to convert the user’s guess from ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54120 245
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement