fklement

fklement

Most efficient way to check for existing data entries

Hello!
I’m currently having a business case where I periodically (about every 5-10 seconds) will need to check if the data that gets inserted does exists in the database. It will also be mostly the case that indeed the data was inserted before.
Now I’m questioning myself what would be the most efficient way to handle that.

Here some solutions I got up with:

  1. *Using |> unique_constraint(:device_id) in the changeset

  2. *Setting an index on the unique identification criteria (every tuple to be inserted has it’s own device_id)

  3. *Trying to fetch data with the given identification criteria (device_id which is known in advance)

  4. *Keeping a state with all currently in the db existing device_ids and checking against that

Would be cool if someone can give an elaborate explanation.

Most Liked

rjk

rjk

As far as I understand the context/your use case,

I would definitely go with the unique_contraint and index on the table on the device_id column to have the most simple and correct way of lookups for these devices. (first make it work and correct)

To go faster (after making it work and correct) you could deploy various techniques from very simple to more complex (giving more headaches because of higher risk of race conditions)

Simplest in my mind would be a cache with a TTL(time to live) for every record you lookup, so you’ll always get a database hit on the first try and then it ‘sticks’ for x seconds or minutes within your cache. This is only applicable if it’s allowed to let a record / device access? linger after a deletion for max TTL timeout. If this is totally now allowed then you need some mechanism to update it after a deletion (See postgres pub/sub listen/notify below)

Now if you want to make it really fast, even on a first lookup, you could create a simple map (fast lookup) on the elixir side if you don’t expect too much of them (else it would take up too much ram), now to keep this up to date you could refresh it completely every x minutes (again, depends on how big the dataset is), otherwise you could use notify/listen (if you’re using postgres) to do a very lightweight pub/sub on modifications of your table.
(see for examples here and here)

To avoid data races, best if you always do your writes directly to the database and ignore your cache. The cache should only be updated by your pub/sub mechanism or full snapshot per x seconds. And you could always treat your cache as the true positive case, as in; if you can’t find them in your cache you do the ‘slow path’ of looking them up in the database. In this case you should always be correct and fast if it’s a returning client.

I’ve used the above technique myself to cache access tokens on the elixir side (which can refresh themselves out of the http request response loop) making the API calls in the microseconds because they don’t have to do a roundtrip to the database to check if they are still allowed. In this case it doesn’t matter if somebody’s access get revoked that it takes a couple of seconds before that cache updates itself and if it’s not found in the cache I always make a database call to see if it exists. So it’s really speeding up the case where API clients return often with a call.

Conclusion, it always depends on the context and requirements but always go for working & correct first, time it and see if it’s fast enough because the database will also keep a lot of your data hot in memory (last fetched records and your index most of the time) so it should be fast enough, if not you could always put in a lot more effort :slight_smile:

Last Post!

fklement

fklement

Wow, many thanks for this detailed insight into your opinion!
Really very interesting.

Where Next?

Popular in Questions Top

hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 130286 1222
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement