gdwoolbert3

gdwoolbert3

I’m writing an application that needs to maintain many buffers that flush to a different location-partition combination in a rotating fashion. There are multiple locations with varying numbers of partitions.

In order to manage this, I’m planning on using an ETS table to maintain a set of records (one per location) that basically look like this:

{"location_key", max_partition, last_partition}

In the above example, max_partition is the largest possible partition index of the given location and last_partition is the last partition index of the given location that was written to.

Whenever one of the buffers flushes, I want to increment last_partition using max_partition as a threshold. I’m aware that there is a function that does almost exactly this but, as far as I can tell, I would have to provide the threshold when I make the call. Obviously, I could first fetch max_partition and then use :ets.update_counter/4 but the trouble is that max_partition can be updated (albeit rarely) so splitting the fetch and update operations opens the door for race conditions.

So, my question is: Is there a way to do all of this in one operation?

(As an aside, I’m also aware that this problem would be technically easier if I used a GenServer, but this will be a very high traffic table so the performance of a GenServer likely won’t cut it in production)

Showing Posts 1 to 3

dimitarvp

dimitarvp

I would attempt :ets.select_replace but that only guarantees atomicity and isolation for a single object read + replace, not everything that matches.

Failing that, you can just use Cachex because it has what you need i.e. transactions.

cevado

cevado

as @dimitarvp pointed out you need a transaction. if you don’t want to add additional library, you can use either :global.trans/2 or :global.set_lock/1 and :global.del_lock/1.

lud

lud

A GenServer is fine if you do many reads and not so many writes.

A GenServer per-buffer could also work well, if you do not have too many buffers.

— All posts loaded —

Where Next? Top

Trending in Questions Top

Blokh
Hey guys, I’ve got a huge CSV ( around 10 GB ) that needs to be processed hourly Do you guys have any suggestions what is the best prac...
New
kszambelanczyk
Hello! Could someone please give me a help/sample code, how to delete a file from s3 using waffle/waffle_ecto from Phoenix app. I creat...
New
Onor.io
I have what I’ve heard referred to as a “lookup table” in my database. This is a way of assigning codes to common values. One common lo...
New
jaybe78
Hello, I’m developing a online persistent chat system (what’s app) like using elixir/dynamodb/aws for a mobile app(flutter). The diffic...
New
Trolleger
What approach to take when sending live updates to “random” users Hi! I have a question, I have a little chat app, and when I create a DM...
New
matt-savvy
Anyone here using Honeybadger? My Honeybadger account is being overwhelmed with noise from some bots. Seeing a lot of Bandit.HTTPError...
New
RemyXRenard
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
New

Other Trending Topics Top

garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
wintermeyer
There are three potential reasons for members of this forum to have a look at https://vutuv.de You are tired or annoyed of LinkedIn. Yo...
New
webofbits
Aludel - LLM Evaluation Workbench Aludel is an embeddable Phoenix LiveView dashboard for evaluating and comparing LLM prompts across mult...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews