Fl4m3Ph03n1x

Fl4m3Ph03n1x

Make atomic updates on ETS key

Background

I have an ETS table where several processes can write concurrently. This table is very very requested, so having a GenServer as a gatekeeper serializing writes is not an option (it can’t hold the load).

Problem

The solution to this, given the GenServer limitations, would be to have each write be atomic. Now, I am aware of update_counter, but there is one problem. My ETS table doesn’t save counters, it saves lists of urls:

  def save_failed_request(url) do
    urls = :ets.lookup(__MODULE__, :failed_urls)

    case urls do
      []  ->
        :ets.insert(__MODULE__, {:failed_urls, [url]})

      [failed_urls: list] ->
        new_list = [url] ++ list
        :ets.insert(__MODULE__, {:failed_urls, new_list})
    end

    {:ok, :saved}
  end

This is the code I currently have to save an URL. Since multiple processes can be calling this function concurrently, multiple processes can be writing in the ETS table under the same key, which is not good.

Question

Is there any ETS function that allows me to make atomic updates on values stored in the table?
Or do I have to use counters ? (because the only atomic operation that exists for updates is update_counter?

Marked As Solved

LostKobrakai

LostKobrakai

Then is really going to be a :bag

tab = :ets.new(name, [:bag])
:ets.insert(tab, {:failed, url1})
:ets.insert(tab, {:failed, url2})
failed = :ets.take(tab, :failed) |> Keyword.values()
# [url1, url2]

Also Liked

peerreynders

peerreynders

:bag comes with the implementation overhead of checking for duplicates on insert; so unless you explicitly need to prevent duplicate full records, you should use :duplicate_bag over :bag .

i.e. may be worth dealing with duplicates after take/2

LostKobrakai

LostKobrakai

It might be better for performance to us :duplicate_bag, which is mentioned to be faster on inserts, and deduplicate the urls after using take/2.

peerreynders

peerreynders

Precedent: https://youtu.be/8mXqxBBvNdk?t=1082

Possible problem: Deleting entries (e.g. take) in ETS can be potentially slow (relatively speaking), it’s faster to drop the table and recreate it.

Solution tradeoff: insert into table can fail temporarily. So errors must be caught and insert must be retried later - this can be potentially delegated to a short-lived task process if inserting process cannot afford to be blocked.

  • rename table with rename/2
  • recreate new table under common, known name
  • do with the renamed table what needs to be done - for example, entire contents can be retrieved with :ets.tab2list/1, :ets.match_object(tid, {'$0', '$1'}), (don’t know performance relative to :ets.lookup(tid, :failed) ).
  • when done delete/1 renamed table

ets-playground
cxy_cache.erl

Last Post!

peerreynders

peerreynders

Precedent: https://youtu.be/8mXqxBBvNdk?t=1082

Possible problem: Deleting entries (e.g. take) in ETS can be potentially slow (relatively speaking), it’s faster to drop the table and recreate it.

Solution tradeoff: insert into table can fail temporarily. So errors must be caught and insert must be retried later - this can be potentially delegated to a short-lived task process if inserting process cannot afford to be blocked.

  • rename table with rename/2
  • recreate new table under common, known name
  • do with the renamed table what needs to be done - for example, entire contents can be retrieved with :ets.tab2list/1, :ets.match_object(tid, {'$0', '$1'}), (don’t know performance relative to :ets.lookup(tid, :failed) ).
  • when done delete/1 renamed table

ets-playground
cxy_cache.erl

Where Next?

Popular in Questions Top

mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New

Other popular topics Top

nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44139 214
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New

We're in Beta

About us Mission Statement