gdwoolbert3

gdwoolbert3

I’m writing an application that needs to maintain many buffers that flush to a different location-partition combination in a rotating fashion. There are multiple locations with varying numbers of partitions.

In order to manage this, I’m planning on using an ETS table to maintain a set of records (one per location) that basically look like this:

{"location_key", max_partition, last_partition}

In the above example, max_partition is the largest possible partition index of the given location and last_partition is the last partition index of the given location that was written to.

Whenever one of the buffers flushes, I want to increment last_partition using max_partition as a threshold. I’m aware that there is a function that does almost exactly this but, as far as I can tell, I would have to provide the threshold when I make the call. Obviously, I could first fetch max_partition and then use :ets.update_counter/4 but the trouble is that max_partition can be updated (albeit rarely) so splitting the fetch and update operations opens the door for race conditions.

So, my question is: Is there a way to do all of this in one operation?

(As an aside, I’m also aware that this problem would be technically easier if I used a GenServer, but this will be a very high traffic table so the performance of a GenServer likely won’t cut it in production)

Showing Posts 1 to 3

dimitarvp

dimitarvp

I would attempt :ets.select_replace but that only guarantees atomicity and isolation for a single object read + replace, not everything that matches.

Failing that, you can just use Cachex because it has what you need i.e. transactions.

cevado

cevado

as @dimitarvp pointed out you need a transaction. if you don’t want to add additional library, you can use either :global.trans/2 or :global.set_lock/1 and :global.del_lock/1.

lud

lud

A GenServer is fine if you do many reads and not so many writes.

A GenServer per-buffer could also work well, if you do not have too many buffers.

— All posts loaded —

Where Next? Top

Trending in Questions Top

stjefim
Hello! Suppose you are building workflow (order / task / payment) processing system with the following requirements: Each workflow con...
New
water
I’m aware of 2 such projects, are there others? git: GitHub - team-alembic/realworld: A fullstack Phoenix LiveView application with back...
New
itzmidinesh
Hi all, I’m working on a Phoenix project where I need to make external HTTP requests and I’m trying to choose between Req and HTTPoison....
New
peck
Hi everyone. My team and I have been working on a fairly modest app based around video streaming and chat, but we’ve landed a customer t...
New
leoschweizer
Hi, we run different LLM models across a cluster, with each node loading a configurable subset. Model-specific queues let nodes consume ...
New
csmr
Hi all, I’m new here and would like to hear peoples views on my Elixir/Phoenix based game-engine! :smiley: Mutonex gamedesign was finish...
New
perzanko
Do we have maintainer of absinthe-graphql/dataloader here? We have a PR that we’d like to merge to the library. Relatively small additive...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
KristerV
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
webofbits
With AI doing more of the implementation work, I’ve been wondering how much coding I should deliberately keep doing myself. My main conc...
#ai
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews