thinkingcat

thinkingcat

Offline-first sync on client -- library?

Hi, I am looking for a library for Elixir that helps to do offline-first type sync to the client. We are currently using Ecto and Absinthe but I don’t think subscriptions can help with asynchronous updates. We can do Listen/Notify, but I haven’t looked into that yet. Hoping there is someone out there who’s thought of this issue already and is working on it as there are a number of good alternatives in other languages :laughing:

We are running Flutter, and this is an example of a synchronization library that is available in Python:

https://github.com/mobync/python-server

Examples in other frameworks are Watermelon DB, Realm and Parse.

Research in Past forum posts have shown me something like Syncing deeply nested list of maps with client, which is very much ‘in the weeds’.

I’m looking for, as I am sure others might like, an easy plug and go library for syncing the database of a offline-first client to the server.

Thanks! :slight_smile:

Most Liked Responses

mdwaud

mdwaud

The Martin Kleppmann suggestion is great - his book is excellent as well.

I work on a similar problem where we have distributed devices running Elixir and Nerves that have intermittent connectivity, but need to reliably transmit data to and from the cloud. The approach I ended up taking was very similar to Kafka or database replication, where data is written to a sequential log on either end and the log is asynchronously replicated between client and server with at least once delivery. This approach solves many problems (reliable delivery, easy to audit or reconstruct system state, etc.), but introduces new ones (linearizability, immutable data, write amplification, etc.). Building on top of this you can use CRDT type approaches like a PN Counter to make eventually consistent values. Beware, though, distributed systems can be a real pain in the neck.

The system runs in production quite well and there is an example app here, however I am sharing this mostly for educational purposes since the protocol is still changing I’m not sure how much support I can give.

It’s a really interesting problem, though, and I’m happy to let you bounce any ideas you have off of me.

ruslandoga

ruslandoga

Subscriptions, listen and notify all require network and wouldn’t help you when the user is offline. Offline-syncing is very much depends on complexity of your data structures, how conflicts are to be resolved, whether you want to only push offline writes from a single client per object (easy), or you need to be able to resolve conflicts between multiple clients to the same object (hard), etc. I’m not aware of any off-the-shelf solutions in elixir ecosystem. There are some CRDT and the recently released Delta which seems to be OT libs out there but any of them might be not suitable in depending on your usecase.

Off-topic – there are really good lectures on youtube by Martin Kleppmann on this and related topics.

Where Next?

Popular in Questions Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 53690 245
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New

We're in Beta

About us Mission Statement