lessless
There is a document export worker with exponential backoff: if a network error occurred, for example, it will retry operation in 35 sec, 246 sec, 1027 sec and so forth.
The worker is built on top of GenServer and documents are plain maps.
Naturally, this opens the door to a race condition when an operation started earlier can overwrite most recent changes.
Req A +---------> Failure +-------------------------------> Success
Req B +--------> Success
It’s worth noticing that insert and create operations are the same HTTP PUT operations and have no difference whatsoever.
To solve this problem I want to introduce operation merging, so when new upsert is scheduled:
- It will check for presence of any worker operating on the same document (thanks for recent Registry release!)
- If there is one and it’s a delete operation, it will be jut stopped
- If there is one and it’s an upsert operation too then:
- Its document will be fetched and merge with the new one
%{old_doc | new_doc} - It will be stopped
- A new worker with updated document will be spawned
In a case of scheduling delete operation scenario is a little simpler: if there is a worker operating on the same document it will be just stopped.
Is this a good plan or is it overcomplicated/oversimplified? What might I be missing?
And advice will be appreciated as well.
Trending in Questions
Other Trending Topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #code-sync
- #discussion
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #elixirconf-us
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #elixirconf-eu
- #processes
- #security
- #api
- #forms
- #performance










Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
minhajuddin
Two options come to mind
brainbag
Depending on how often you expect your update request to fail and how important the ordering is, you can queue the results.
If you give each request an upsert sequence number (like a TCP packet) per document, you can check when a new result comes back to see if it’s out of sequence, and if it is, queue the write operations (or whatever) until the out of order request has returned.
If it’s possible for you to drop upsert requests because it fails for too long, you can then remove requests from the queue once it passes a certain threshold, and send the request originator an error message.