Hi @bartblast I am super excited to use Hologram and started building my application in Hologram.
For pages that I dont need realtime updates, I am able to use Holograms but for ones that need realtime updates i am unable to use hologram am I missing something here.
Is there support for Realtime serverside updates like LiveView Updates, Phoenix Sync and support for Live Data Streams from Server side like in LiveView ?
Use-case for live chat applications and live dashboards update.
If the support isnt there yet what would you suggest for such usecases ?
PhoenixSync is built on top of ElectricSQL, which parses the Postgres WAL and streams subsets out to clients. You can use the Electric client directly to stream updates. I donât know Hologramâs APIs well but I believe there is some way to push those updates down to the client. Then you can materialize the view yourself client-side.
There are also some (me) working on databases that natively support partitioned WAL streaming, so that will be an option someday!
Hi @krezicoder!
Great to hear youâre building with Hologram!
Youâre not missing anything - currently, data from the server can only be sent in response to commands. Server-initiated updates arenât supported yet, but there are two features on the roadmap that directly address your use cases:
Server-Triggered Actions - Enable triggering actions from server code outside of commands, such as background jobs (e.g. Oban) or scheduled tasks.
Local-First Support - Implement local-first architecture with offline-capable local database, auto-sync, and conflict resolution for resilient applications.
Initially I planned to provide a PubSub API, but after deep exploration and Time to Implement Pub/Sub in Hologram - Looking for Your Input!, I shifted toward a local-first auto-sync approach - along the lines that @garrison advocated in that thread. The key insight: using PubSub for data sync forces you to manually wire up message passing and maintain synchronization yourself. This leads to bugs and tangled imperative synchronization code. A local-first layer gives you declarative, automatic sync - data just appears in your components when it changes, no manual routing needed. You want top-down data flow from a central source of truth, not component-to-component messaging.
Weâd still have Server-Triggered Actions for ephemeral events (notifications, typing indicators) where they make sense - playing a role similar to cast (fire-and-forget, no delivery guarantee).
The local-first space is enormous and Iâve been analyzing it for some time, hence the silence on this topic.
What you can do today
With some creativity, you can support your use cases now with a combination of these:
Polling command - a command that periodically checks for updates from the server using the delayed param
Phoenix Channel alongside Hologram - store channel events in a global JS object, then read from it via Hologram.JS.exec/1 in a polling action
Hologram.JS.exec/1 for general but basic JS interop
Iâm all for syncing but what if youâre not using ElectricSQL (f.e. because you donât have Postgres).
Iâm looking into hologram for âkanbanâ like board where each task is separate Genserver.
Initially Iâve had this written with Postgres+Electric+Phoenix+Separate FE, but after giving it some thought Iâve dropped PG in favor of SQlite (because Iâll usually be running it locally anyway) so that means Iâve lost access to PhoenixSync and itsâ niceties.
Generally polling wonât be good enough for this usecase in a long run so it would be nice to have some recommended solution that wouldnât necessarily tie you into syncing (I guess in my particular case I wouldnât care if Phoenix Sync had SQlite backend ;-
EDIT. depending on how this âsyncâ layer would be implemented, if we could plug into it ourselves somehow (so I can write code responsible for âsyncingâ Genserver) that would work for me too.
Hi @nxy7! I canât say yet exactly how it will look because this part of Hologram is still in the research phase. It may use ElectricSQL under the hood, or it may have its own low-level solutions - Elixir is very well suited for writing this kind of orchestration, by the way.
Regardless of how itâs implemented in the end, the goal is to be declarative - you wonât have to do all this plumbing yourself, that will be Hologramâs job. Youâll only define some rules and the rest will be done automatically. For example, youâll be able to use the data client-side, save it, and it will auto-sync with the central data source (or be kept client-only if you define it that way).
One thing worth considering: having âeach task as a separate GenServerâ sounds like you might be thinking in the LiveView paradigm, where state lives server-side. Hologram works differently - code executes client-side in the browser. Your tasks would typically be client-side data structures that sync with a central source of truth, rather than individual server processes. The sync layer will handle keeping everything consistent across clients.
Well in my case itâs a requirement for tasks to be executed server side and for clients to just have a view of the data (thatâs why I was thinking of it in terms of syncing server state into clients).
In my app once the task is moved into some column it starts execution pipeline which happens on the server (start some processing/send notifications/etc etc, all of that ideally with realtime notifications, streaming text in task details and showing current processing status).
I know that Hologram is focused on the client more, but I hope that itâll also have some answer to keeping server state in sync with client too In truth I was thinking that Hologram isnât even âclient centric frameworkâ but âisomorphic frameworkâ thatâs aiming to be able to work with both âserver heavyâ and âclient heavyâ approaches.
I love that youâre going for declarative approach everywhere btw.
Iâm really hoping that youâll manage to think of more generic solution that wonât require Postgres (but thatâs just my selfish wish, even if it does I can make use of it in other projects) because that might open up some interesting opportunities and maybe could avoid some bottlenecks associated with Electric (depends on single DB and might not be suitable for ephermal data as far as Iâm aware).
Sync is not trivial; there are certain guarantees the database has to be able to make.
In addition to ElectricSQL (for Postgres) there is also @jstimps 's work on EctoFDB Sync, and then thereâs my work on Hobbes which is more foundational (implementing a DB from scratch).
I would suggest @bartblast try to come up with a design that is general enough to support any backend, but that may actually be somewhat difficult. For example, Postgres can be incrementalized (ElectricSQL does this with WAL parsing) but FoundationDB on the other hand cannot. This concern leaks into the framework layer because if Hologram runs on the client you want to maintain the view on the client, so it canât be oblivious.
My hope is that Hobbes can provide an Elixir-native database primitive here (with proper incremental support) so that framework designers like Bart do not have to worry about database correctness stuff (which can get pretty in-the-weeds Iâm afraid). I am making good progress on that front. Soon!
I wasnât suggesting that sync is trivial, itâs obviously super hard to get right. I just hope that making âgenericâ API for sync layer is easier than sync itself so that hologram would allow for pluggable backends
My initial thought was that it might be easy to narrow the problem down and start with keeping some server process state in sync with client and then multiple backends can sync state in there, so how that backend process is being kept in sync with itâs source is someone else problem and Hologram would just take care of syncing this process with client (this would immediately allow usage of ElectricSQL using itâs Elixir API).
It obviously has drawbacks (like maybe you donât want to keep this state materialized on the server) but thatâs something that would be perfect for my usecase (I actually want to have tasks running on backend and show their state in real time in hologram without much effort - right now Iâm doing it using Phoenix PubSub).
EDIT. just read through hobbes README and it sounds super interesting for my usecase since Iâm already using embedded DB. Iâll keep an eye on it
EDIT2. now that I think about it having a way to keep BE state in sync with FE without involving DB at all is actually super relevant for something like Hologram. Since Elixir is used for a lot of realtime stuff it seems reasonable to want to use realtime things from existing ecosystem (like Presence) and it seems to be smaller problem to tackle than going all the way to keep FE in sync with DB state.
You could come at it from the perspective of âhow do I avoid the DBâ, or you could come at it from a perspective of âhow do I design a DB that works well hereâ. In practice many (though not all) of the things you might think belong outside of the DB actually should be inside of it, but because of the way RDBMS are designed they add a lot of friction.
There have been many attempts to fix this over the years. Probably the most promising was Firebase, but unfortunately it was acquired by a company that hates building products.
Having spent a lot of time on this topic, Iâm just not sure this is actually the case. For deeply technical reasons the sync part is actually quite difficult to separate from the database. I think a more promising path is to design a database that supports sync with a very open-ended data model. I believe I can make it work, but itâs hard to say for sure.
I think that no matter what there will be need for some sync/realtime communication primitive not involving DB because otherwise youâd not be able to work with big part of the ecosystem (like Presence) and existing tech.
Even if this primitive would not give as many capabilities as some âblessedâ solution (like some nice DB integration) right?
Because otherwise weâd be saving things to DB just to get sync and not persistance. Iâve seen some of your posts and know that very likely You have this thought out better than me so Iâll ask - do you think that with good enough DB solution (maybe something built on top of Hobbes that youâre developing) it would be practical to push everything you want to sync into DB? Presence / cursor positions / realtime stock prices etc etc.
All of those are very simple data (not derived data or some aggregates), so some solution that would just be synchronizing state kept on the server with client would suffice but as youâve said my intuition is heavily influenced by RDBMS so I might have completely wrong idea about it and maybe DB might be a good place for ephermal data too.
You are not the first person to ask me this question; Bart asked me almost exactly the same thing in a past thread. My answer is still no, I would not persist cursor positions to the database. I would, however, probably persist more things to the database than you would think. That pipeline stuff you were talking about sounds like prime database territory to me (though of course I donât know your case in detail).
I believe there is a need for a realtime communication primitive that is not sync, and I think thatâs what youâre getting at. However, I think that primitive is message passing.
Sync, to me, implies certain guarantees about consistency. You do not want events to arrive out-of-order, for example, or for some events to have gone missing. The trickier one is that itâs important that everything that is supposed to be âavailableâ at a given timestamp is actually there or you will get very nasty bugs. This is difficult to achieve without essentially inventing a database from first principles (and that is exactly how I ended up here).
So, to be clear, I am not disagreeing with you. Iâm just not calling âthat thingâ sync. I think what you would want is an extremely broad escape hatch that mirrors message passing, i.e. something very close to the raw WebSocket connection. A push_event or similar. I believe there are plans for Hologram to have such an API.
If the state is small and performance can be manged by a single GenServer, do you still need a database? I agree true state syncing is much more interesting and useful than message passing.
This very much depends on context. There are tons of things for which a GenServer is exactly what I would recommend, e.g. for the ârealtime cursorsâ example.
But if you have transactions over application data things start to get tricky. Itâs very important that the âlogâ of events youâre consuming is designed properly. If you can observe transactions out of order, or partially, things go off the rails quickly. The more stuff you do outside of the database, the more room there is for weird consistency bugs.
OTP is very powerful, but itâs also pretty old and lacks good primitives for building distributed systems with strong consistency. Hobbes basically only uses message passing, local name registration, and ETS. Tools like :mnesia, :global, and even monitors are useless in practice.
Also, for a lot of stuff you actually want persistence, and even just safely writing data to disk is so much harder than you probably think. Thatâs why Iâm trying to build a primitive that can take care of that stuff. I think I can lower the barrier to entry substantially.
You can implement state sync over message passing this way, and itâs fine for many things. But the polling increases the latency, so eventually you will start to get ideas about how you could push the state updates to the client directly and you will reinvent the log from first principles as well
Producing a delta is also not trivial in practice. You would need to diff the current version against a past version, which means your state is now a persistent data structure. Now you need to invent garbage collection. And how are these objects versioned? Are you guaranteeing linearizability? Etc.
Itâs true that EctoFDB itself doesnât attempt to incrementalize FDB state, but Iâm not convinced we can say that it cannot be done. FDB can take consistent backups, both snapshot (backup agent) and real-time (DR agent). Someone may be able to tap into that system in a novel way. I tried, but ran out of time and focus, and abandoned that effort. It does require reverse engineering the data stream, which is not documented, and could change in any new version. Maybe that is enough to call it impossible.
On the other hand, according to my favorite ghost in a jar (Gemini), Postgres does officially support low level WAL parsing, so kudos to them for that.
Will Hobbes incrementalization be based off of a read version? e.g. the client is notified of the content of a change at a particular version stamp, and the client is responsible for understanding the full state at that precise version stamp so that it can consistently process the delta? Or will the communicated delta provide a from-versionstamp and a to-versionstamp, allowing the client to evolve their local state to the new versionstamp?
You can definitely modify FDB and add a new role to parse WAL segments (this has been done before for streaming backups). Parsing the backup output from the DB itself is an interesting idea but I would expect there to be a lot of latency. I could be wrong.
Thereâs also the fact that mutations are spread semi-randomly onto TLogs due to FDBâs frankly unhinged replication policy. If you check the Hobbes repo I actually just wrote a new RFD about that because Iâm finally moving to copysets. Gonna be some messy diffs today lol.
My idea is to allow a subscription to a key or range on a given Storage server. The Storage server already applies its mutations in perfect version order, so it can simply check the list of subscriptions as it goes.
Itâs important that you can perform a read+subscribe without missing anything in between, and we can support that by performing the subscribe at the same read_version as the reads. At the time the subscription is opened the Storage server can read the multiversion store for that range and send logs for any keys that were modified afterread_version immediately. I assume FDB watches must do something similar.
I donât see any reason to send âfromâ versions. The primitive is âgive me all changes for this range and do not miss anyâ. As long as you actually fulfill that promise, you can always reconstruct a valid view of the range.
There is one more edge case: you have to send nacks to the client when there are no mutations at a version. Otherwise you canât maintain a view across shards because the frontier wonât advance.
In truth I was thinking that Hologram isnât even âclient centric frameworkâ but âisomorphic frameworkâ thatâs aiming to be able to work with both âserver heavyâ and âclient heavyâ approaches.
