Offline-first sync on client -- library?

thinkingcat

Hi, I am looking for a library for Elixir that helps to do offline-first type sync to the client. We are currently using Ecto and Absinthe but I don’t think subscriptions can help with asynchronous updates. We can do Listen/Notify, but I haven’t looked into that yet. Hoping there is someone out there who’s thought of this issue already and is working on it as there are a number of good alternatives in other languages

We are running Flutter, and this is an example of a synchronization library that is available in Python:

https://github.com/mobync/python-server

Examples in other frameworks are Watermelon DB, Realm and Parse.

Research in Past forum posts have shown me something like Syncing deeply nested list of maps with client, which is very much ‘in the weeds’.

I’m looking for, as I am sure others might like, an easy plug and go library for syncing the database of a offline-first client to the server.

Thanks!

3 comments

#ecto #deployment #synchronize

1 1805 3

2023-05-22 23:46:55 UTC

First 3 of 3 Posts

ruslandoga

Subscriptions, listen and notify all require network and wouldn’t help you when the user is offline. Offline-syncing is very much depends on complexity of your data structures, how conflicts are to be resolved, whether you want to only push offline writes from a single client per object (easy), or you need to be able to resolve conflicts between multiple clients to the same object (hard), etc. I’m not aware of any off-the-shelf solutions in elixir ecosystem. There are some CRDT and the recently released Delta which seems to be OT libs out there but any of them might be not suitable in depending on your usecase.

Off-topic – there are really good lectures on youtube by Martin Kleppmann on this and related topics.

Post #1

thinkingcat

Thanks so much @ruslandoga!

Is there something that we can model on in Elixir from other languages?

I put this matter off for the past year or so (as you can see) but I will be back on it soon. Have a great one!

Post #2

mdwaud

The Martin Kleppmann suggestion is great - his book is excellent as well.

I work on a similar problem where we have distributed devices running Elixir and Nerves that have intermittent connectivity, but need to reliably transmit data to and from the cloud. The approach I ended up taking was very similar to Kafka or database replication, where data is written to a sequential log on either end and the log is asynchronously replicated between client and server with at least once delivery. This approach solves many problems (reliable delivery, easy to audit or reconstruct system state, etc.), but introduces new ones (linearizability, immutable data, write amplification, etc.). Building on top of this you can use CRDT type approaches like a PN Counter to make eventually consistent values. Beware, though, distributed systems can be a real pain in the neck.

The system runs in production quite well and there is an example app here, however I am sharing this mostly for educational purposes since the protocol is still changing I’m not sure how much support I can give.

It’s a really interesting problem, though, and I’m happy to let you bounce any ideas you have off of me.

Post #3

— All posts loaded —