ddoronin

ddoronin

Trying to understand gc_interval in Nebulex

Hey folks! I’m trying to understand gc_interval in Nebulex.

# GC interval for pushing new generation: 12 hrs
gc_interval: :timer.hours(12),

I know how generations work in .Net and JVM, but I struggle to understand what that parameter means and how I should choose the optimal timeout?

Let’s say I have a cache of short-living objects with TTL 5mins, what is the best gc_interval?
Another case, I cache the HTML file, the TTL is 1 day, what is the best gc_interval?

Marked As Solved

ruslandoga

ruslandoga

I checked nebulex source code, and it’s a purely garbage collection concept. One way to do garbage collection (i.e. deleting stale values from the cache backed by an ets table) is to create a timer for each new entry and delete them when the timer fires. Another is to run a cleanup process that deletes all stale values in one big scan every other interval. And then there is the way nebulex uses, which is to have multiple ets tables and release whole tables when the time comes. So every gc_interval the following happens:

  • tables before garbage collection: [ets1, ets0]
  • gc_interval timer fires, a new ets table is created [ets2, ets1, ets0]
  • oldest table is dropped [ets2, ets1], since nebulex seems to keep at most two tables at a time, all entries of ets0 are now “garbage collected”

So the value of gc_interval seems to depend on how long you can tolerate stale values occupying memory on one hand (so it should be as little as possible after ttl expires) and on the other hand tables shouldn’t be deleted while there are active entries in them so gc_interval >= ttl (you can cache a value right before gc_interval timer fires so lifetime(old_ets) should be at least ttl).


IMHO this approach might be overkill in some scenarios. Personally, I’ve never had issues with “naive” cleanup techniques.

Where Next?

Popular in Questions Top

_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
vac
Hi, I’m quite new in Elixir and I’m trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and I...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> somethi...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New

Other popular topics Top

chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36128 110
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New

We're in Beta

About us Mission Statement