Time to Implement Pub/Sub in Hologram - Looking for Your Input!

Well … if you would implement something like PubSub then people would have to write wrappers for simplest things like send/2 and Process.send_after/3. However actions and commands would loose it’s meaning if you would implement handle_* support.

How about we would have 2 PIDs in server and client state? One goes for server and one for client. Then we can simply call send and Process.send_after on those pids and use some DSL functions similar to actions and commands to handle them.

With that any time like on initialisation or in action/command we would be able to start a background process in either client or server that would send it’s results for server, client or both depending on needs. That would be very flexible.

Interesting idea… Could you share some code snippets to help explore this?

We’re focusing on the ideal developer experience right now - internal implementation details can come later. I’d love to see:

• What a simple example might look like (chat message, live counter, etc.)
• How the app would keep track of these PIDs (per component? globally? lifecycle management?), etc.

Just pseudocode is fine - want to get a feel how this would play out from a DX perspective.

Actually I forgot about components … Ideally there should be 4 ways to get PIDs of 3 types.

A server PID would be a JS equivalent for Elixir process that works like a proxy with server process. The message would be send via WebSocket.

A client PID would be a JS equivalent for Elixir process that works like a proxy with client process. The message would be send to current page (client side).

A component PID would be a JS equivalent for Elixir’s self() process. We simply want to send message to current component (anonymous function, action or command).

The fourth way would be an easy way to get a parent component PID or page PID. This is for “just send to parent whoever it is”. This would be nice feature for a behaviour-like implementations of some actions and commands.

defmodule MyApp.GreetingPage do
  use Hologram.Page
  
  route "/hello/:username"
  
  layout MyApp.MainLayout

  def init(params, component, server) do
    component
    |> put_state(:client_pid, component.client_pid)
    |> put_state(:component_pid, component.pid)
    |> put_state(:server_pid, server.pid)
  end

  def template do
    ~HOLO"""
    <div>To communicate with app use following process identifiers:</div>
    <ul>
      <li>Client PID: {@client_pid}!</li>
      <li>Component PID: {@component_pid}!</li>
      <li>Server PID: {@server_pid}!</li>
    </ul>
    """
  end
end

With that we can use those PID to send a message:

# optional 3rd argument for passing data
holo_send(client_pid, :action_name)
holo_send(server_pid, :command_name)
holo_send(component_pid, {:action, :action_name})
holo_send(component_pid, {:command, :command_name})

# raises no such action error:
holo_send(client_pid, :command_name)
# raises no such command error:
holo_send(server_pid, :action_name)

# same for holo_send_after

Yes, client_pid and component_pid may send same actions as well as server_pid and component_pid may send same commands. This would happen only if component is a page. However all of those PIDs needs to represent different process (or JS equivalent) so that we can send strict actions to client or commands to server and alternatively send generic actions and commands within current component regardless if current component is page or not.

This way client_pid and server_pid would be recommended to use directly within init, command or action and component_pid would be used in more generic cases like external hex packages. Look that some developers may want to use actions as equivalent of JS intervals - i.e. background process that send message to itself (same action name) after specified time and based on some condition based on send data (if any).

I would strongly favor a declarative sync-based API for tracking server state on the client. I don’t know exactly how this would look (nor do I have a great understanding of Hologram’s existing APIs). But using Phoenix terminology, some way to specify that a given set of assigns should be automatically shipped to the client and kept up to date (with no tearing) would be a good path, I think.

I would not recommend venturing down the path of encouraging developers to maintain synchronization with their own imperative “glue” protocol (i.e. sending their own diffs down the wire by hand). This always ends badly.

I think it depends how far you want to abstract things. I think Hologram is in a unique position to make an amazing DX here.

If Hologram’s ultimate goal is to enable local-first apps, what if we started from the ideal end state. I imagine that would look something like this:

Declare reactive live queries that sync into local state. Using a chat app as an example, maybe the syntax would look something like this:

messages =
  sync do
    Message
    |> where([m], m.room_id == ^room_id)
    |> order_by([m], asc: m.inserted_at)
    |> Repo.all()
  end
end

Or

messages =
  Message
  |> where([m], m.room_id == ^room_id)
  |> order_by([m], asc: m.inserted_at)
  |> sync_all() # sync_one also available 

These could be declared in a component or page.

Hologram would automatically create client-side stores (I imagine at the table-level, not for each query, so you don’t have duplicated data). The queries would be kept alive even when leaving the page / component is destroyed for some period (default to 5 min) so that if a user navigates away and then back within that time period the data is still in sync. In the future these could be synced into indexeddb (or OPFS possibly, I only have experience with indexeddb) under the hood but it acts more of a sidecar. Keep active live queries in memory for optimal performance. All of the traditional complexity could be abstracted away.

This would be conceptually similar to Zero and Meteor but the syntax and concepts could be made even more straightforward. From the developer’s perspective, they are just writing queries and everything “just works”. For the first version, I’m not sure Zero’s IVM approach would be needed, could be left as a future endeavor.

Naturally I find this approach very enticing, but I think there is a real risk of complexity explosion and “accidentally building a whole database” here. I mean, what exactly are you querying? Rows? From where? Who is persisting those rows?

You could pare the complexity back a bit by just offering a set of APIs for syncing sets (the assigns like I described) knowing that you could then incrementalize query operations over those sets using something like differential dataflow. This would be much easier to implement and would give Hologram a base for developers to start experimenting on. Then maybe somebody comes along (maybe you!) and adds queries on top of the sets and implements the IVM stuff (“Incremental View Maintenance” for those unaware).

I’m not saying it wouldn’t be better to implement everything from the get-go BTW, it’s just that it would be an enormous undertaking.

Electric sql does exactly that with tanstack/db and d2ts.

This is exactly what I had in mind. I was going to link d2ts and forgot!

Thanks everyone! I love how we’ve accidentally created a buffet of real-time patterns

If I understood correctly:

• @Eiji proposed process messaging (very Elixir-native!)
• @garrison proposed state sync (declarative and clean)
• @jam proposed reactive queries (ambitious and exciting - I’d definitely like to explore the local-first path you showcased! I have something similar in mind for next stages)

All fascinating approaches, but I should clarify - I’m specifically looking for Pub/Sub (Publish/Subscribe) patterns. Think broadcasting messages across channels/topics like Phoenix Channels and Phoenix PubSub, but reimagined for ideal DX.

The exciting part: Hologram could eventually work across clusters of devices (web/mobile/desktop), so we could target specific users, sessions, or even component IDs (cids). We could broadcast to “all users in room X,” “session Y,” or “component Z on any device.”. Eventually… For now let’s just make it work for common cases.

Forget implementation details or what Phoenix did - what would the ideal developer experience look like for publishing/subscribing to messages across this kind of distributed component system?

Let’s get creative!

You know, this probably isn’t the answer you’re looking for but I’m not sure if PubSub is a good idea for exactly the same reasons I mentioned above.

I have experimented quite deeply with building “realtime” apps using Phoenix PubSub in the standard way and I have come to the conclusion that it’s just not good. Using a streaming system to build apps like this is rather fundamentally a bad idea I think. I mean, it can be done right but then you end up with differential dataflow and similar (this is the point of that article, which I recognize is a bit much for this discussion).

The ElectricSQL guys built Phoenix Sync for this reason and that presents a better path. Personally I decided to head down the path of building an entire DB from scratch to solve this problem, which I hope will soon be another helpful contribution in this area for the community.

Given the above, I’m not entirely sure this is something Hologram should do at all. Instead of PubSub, maybe integration with something like ElectricSQL would be better. This stuff is really hard to get right.

Now maybe there are some things better suited to a PubSub system rather than a “streaming database”, but honestly I am actually struggling to think of anything. I’m sure somebody else can come up with an example, though!