rm-rf-etc

rm-rf-etc

Replacing stored procedures with Elixir processes?

Some of the apps where I work frequently suffer outages as a result of our stored procedures, and someone suggested that maybe we could remove some of this load on the database by bringing the data into elixir. Has anyone here seen something like this where they work?

The query-by-id parts would likely get translated into phoenix topics. What would be a good way to backup the in-memory data in case a process crashes? I’m imagining that elixir could generate the needed processes based on demand, so that any processes representing objects that haven’t gotten any attention in a while, could be put to sleep and left in the database, to be warmed up the next time someone asks for those objects. Our apps stall when specific records get a lot of attention, so the reads and writes tend to be concentrated around a specific object ID and all the records associated with it.

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Ah! I finally get it. Which database are you on? This stored procedure is basically a home grown version of postgres’s INSERT ON CONFLICT so if you’re on postgres you may be able to refactor this to just use that and move on.

Beyond that though, I was referring to EXCEPTION WHEN unique_violation THEN. The whole thing is weird though because if the id is a primary key, indexing on (id, month) shouldn’t actually help because querying on just id should already be indexed. Where does arg_id come from in your stored procedure?

mindok

mindok

There’s a discussion on caching to reduce DB load here (particularly for heavy reads that are frequently hit). It is probably a bad idea to roll your own caching system using raw processes - it gets hard quite quickly (e.g. handling timeouts, knowing when to invalidate etc).

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Is data loss acceptable?

This is exactly the issue. If your database is slow and inefficient, batching data in memory before inserts just adds another layer of failure. It isn’t clear how this would even help with the stored procedure issue, cause those will ultimately fire when inserted anyway.

If you have valuable data, and you can’t tolerate data loss, then you can’t afford to store that data in memory, it must be written to a disk somewhere. If reads are your issue and not writes then you can look into caching.

Last Post!

rm-rf-etc

rm-rf-etc

Yup, I see it. I shared this link with the team.

I think I was confused what the IF found does. I think I understand now.

Where Next?

Popular in Discussions Top

AstonJ
Are there any Elixir or Erlang libraries that help with this? I’ve been thinking how streaming services like twitch have exploded recentl...
New
lorenzo
Hey everone! I created a prototype for my app using Nodejs for the api. But the framework I chose wasnt great (in general theresnt any g...
New
chuck
Let me start by stating an assumption: Phoenix is a great approach to building REST APIs. There are many reasons for this, but I will ass...
New
slashdotdash
Phoenix Live View is now publicly available on GitHub. Here’s Chris McCord’s tweet announcing making it public.
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
lucaong
Hello Elixir and Nerves community, I have been working for a while on an open-source embedded key-value database for Elixir, that I call...
230 14362 124
New

Other popular topics Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 44608 311
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 42576 114
New

We're in Beta

About us Mission Statement