Merging multiple network requests

Questions / Help
1

There is a document export worker with exponential backoff: if a network error occurred, for example, it will retry operation in 35 sec, 246 sec, 1027 sec and so forth.
The worker is built on top of GenServer and documents are plain maps.

Naturally, this opens the door to a race condition when an operation started earlier can overwrite most recent changes.

Req A  +--------->    Failure   +------------------------------->    Success

                                  Req B  +-------->  Success

It’s worth noticing that insert and create operations are the same HTTP PUT operations and have no difference whatsoever.

To solve this problem I want to introduce operation merging, so when new upsert is scheduled:

  1. It will check for presence of any worker operating on the same document (thanks for recent Registry release!)
  2. If there is one and it’s a delete operation, it will be jut stopped
  3. If there is one and it’s an upsert operation too then:
  4. Its document will be fetched and merge with the new one %{old_doc | new_doc}
  5. It will be stopped
  6. A new worker with updated document will be spawned

In a case of scheduling delete operation scenario is a little simpler: if there is a worker operating on the same document it will be just stopped.

Is this a good plan or is it overcomplicated/oversimplified? What might I be missing?
And advice will be appreciated​ as well.

2

Two options come to mind

  1. To make things simpler, you could avoid running a second operation when a previous operation on the same document is still running
  2. Use optimistic locking to make sure that you don’t overwrite data (if this is an option and if you have a database)
3

Depending on how often you expect your update request to fail and how important the ordering is, you can queue the results.

If you give each request an upsert sequence number (like a TCP packet) per document, you can check when a new result comes back to see if it’s out of sequence, and if it is, queue the write operations (or whatever) until the out of order request has returned.

  1. Upsert request #140 fails on the first two tries, still waiting on the third
  2. Upsert request #141 fails on the first try, then succeeds on the second
    • but #140 hasn’t returned yet, so queue #141’s result
  3. Continue to handle requests until #140 returns
  4. Write everything in order (or whatever you do with the results)

If it’s possible for you to drop upsert requests because it fails for too long, you can then remove requests from the queue once it passes a certain threshold, and send the request originator an error message.