Realtime collaboration

[tmbb]

After a long time, I’ve decided to go back to realtime collaboration on Phoenix. The plan is to implement something compatible with ShareDB's protocol. There are already some tools using ShareDB in the wild, and the most important case (collaborative editing of text) is already covered by ´ShareDB` pretty well (there’s a demo under 20 LOC doing exactly that with some extensions).

ShareDB has a client and a server implementation, both written in Javascript. In an ideal world, I’d port the protocol to Elixir and reuse the client without requiring any changes. However, the protocol expects messages to be sent through “bare” websockets, and Phoenix expects messages to be sent through “channels”. Channels are ingrained so deeply in Phoenix that I don’t think it’s possible to use bare websockets wthout reimplementing the whole channel machinery. In fact, Phoenix doesn’t even know what an websocket is.

The client-side API for ShareDB starts with something like this:

var ShareDB = require('sharedb/lib/client');
var socket = new WebSocket('ws://localhost:8080');
var connection = new ShareDB.Connection(socket);

The socket really must be a WebSocket, or at least something that implements the websocket methods. That’s actually very easy with Phoenix channels: you just have to create an object (class, prototype, whatever) which implements those methods and behaves as a webocket. However, there is a problem.

ShareDB allows you to create several “documents”, which can be updated in parallel. That means each document should live in its own process and be connected to the client though its own topic. Our “custom” websocket implementation would have to route each message into the appropriate phoenix topic. But the WebSocket object is sent raw JSON and doesn’t know which document/topic the message refers to. The only way to know that is to parse the JSON again and extract the document’s ID, which is unacceptable.

This means I will have to rewrite both the Connection object and the WebSocket object so that the WebSocket is compatible with Phoenix channels and the Connection object can send objects into the WebSocket (so that it can route the messages into the correct topics).

[tmbb]

Ok, I’ve been looking more deeply into Phoenix PubSub and it looks like I can decouple the use of Phoenix PubSub from the client-side channels. This means it’s possible to create a websocket-like channel on top of Phoenix channels. These channels could all connect to the same topic and I could separate them into PubSub topics on the server (at least I think I could do it). I would appreciate help in implementing this websocket-like object on top of Phoenix channels

[tmbb]

I’ve managed to come up with this: https://gist.github.com/tmbb/a723471ce7d1d666289edea058bfd652

It’s a Javascript class that behaves as a a websocket, but which sends and receives all messages over a Phoenix channel. I’ve plugged this Javascript class (XareDBWebSocket) into a websocket chat example I’ve found somewhere on the internet and made it work with a phoenix channel backend instead of raw websockets. This means that the client is basically handled.

Should I publish this “custom websocket” as a node package? I think it could be useful for other people which want to write Phoenix backends for Javascript tools which expect a websocket connection.

[axelson]

@tmbb

Should I publish this “custom websocket” as a node package? I think it could be useful for other people which want to write Phoenix backends for Javascript tools which expect a websocket connection.

I think so. I think I remember someone posting in the Elixir Slack about wanting to use raw websockets and they were pointed to Cowboy, which I think would be more difficult than using your package.

[entone]

The IETF has the Braid protocol https://datatracker.ietf.org/doc/html/draft-toomim-braid, which I believe is used by SharDB, or something similar, would be super interesting to see something like this integrated into Elixir/Cowboy/Phoenix

And here’s the website, https://braid.news/

[entone]

It was specifically this that caught my attention

Braid allows Synchronizers to Interoperate
Rafie Walker has used the Braid protocol to synchronize an OT system with a CRDT. A babelfish sits in the websocket, and converts ShareDB’s network messages into Braid messages, and vice versa.

[tmbb]

Back from the dead. As part of the plan of writing a ShareDB backend in Elixir (so that we get all the real-time collaboration features for free), I’ve had to solve the problem of protocol mismatch between Phoenix and the ShareDB client. The ShareDB client is supposed to work with raw websockets, while Phoenix can only work in Channels. You can’t use raw websockets with Phoenix without getting yourself in a world of pain.

The obvious solution is to implement a websocket on top of Phoenix channels. If this makes as little sense as it made to me at the time, read on. The trick is to write a Javascript object that implements the websocket API but sends and receives messages over a Phoenix Channel.

I’ve ended up with something like this: https://gist.github.com/tmbb/d0a4930cef8dc41a0125736fb6e0fb6e

As you can see there, I build a PhoenixWS, which is an object that talks like a websocket but sends and receives messages over a Phoenix channel:

const connection = new PhoenixWS(socket, "room:lobby", {});

It takes as arguments a phoenix socket, a channel topic and an initial payload (which is useful for authentication, for example). This is the only phoenix-specific part of that file. From that point on, the javascript code is written as if it were a normal websocket. The rest of the code in that file was actually copied with no changes from a random Javascript “chat server” implementation, meant to work with a websocket on top of NodeJS. This chat server now works with a Phoenix backend (see here: https://github.com/tmbb/phwocket_example)

The implementation is not complete and this is not production-ready yet, but it shows how easy it is to achieve some basic compatibility with raw websockets while being able to use Phoenix channels underneath.

The Elixir side is actually quite easy too. You add your PhoenixWS.Channel to your socket:

defmodule PhwocketExampleWeb.UserSocket do
  use Phoenix.Socket
  require PhoenixWS.Socket

  ## Channels (actually a very weird channel that talks like a websocket)
  PhoenixWS.Socket.channel("room:*", PhwocketExampleWeb.RoomWSChannel)
  # ...
end

The code that implements the websocket channel is actually pretty easy to write:

defmodule PhwocketExampleWeb.RoomWSChannel do
  # Import some convenience macros to abstract away the implementation details
  use PhoenixWS.Channel, web: PhwocketExampleWeb

  def join("room:" <> _, _payload, socket) do
    {:ok, socket}
  end

  # Handle messages from the Client
  def phoenix_ws_in(data, socket) do
    # This is just an echo server, so we just broadcast the same message back into the client
    broadcast!(socket, data)
    {:noreply, socket}
  end
end

Although there are still some holes in the websocket implementation (it doesn’t close properly, for instance), this can serve as a bridge between the ShareDB client and a backend that runs behind Phoenix. I imagine that the PhoenixWS package might be useful by itself in projects where you want to interface directly with a Javascript library that uses websockets.

Expect more developments soon, but not thaaat soon.

[Exadra37]

Sorry about my ignorance but looking to the ShareDB at a glance in Github it looks likish Phoenuix Live View, but I may be wrong, and probably I am.

So what you are trying to achieve it’s not possible through leveraging Live View?

[axelson]

@tmbb

Back from the dead.

Welcome back!

ShareDB backend support in Elixir would be quite neat, and since it uses Operational Transforms it should allow some support for offline use-cases (@Exadra37 which LiveView definitely doesn’t support).

Also I wanted to say that the proxying approach to support raw websockets over Phoenix Channels is very clever. It would be nice to extract that to it’s own URL in some fashion (whether it is a blog post, separate forum post, or github repo, etc). I’m pretty sure that other people have asked about it and usually they’ve been told to drop down to cowboy which is not that ergonomic.

[TardisBlue]

@tmbb any updates on this? I am working on a project that uses ShareDB (along with a custom OTType) and it would be great to try and use it while leveraging Pheonix Channels in the backend with 1 channel per document room. If something like this is in the works I would be willing to help out, although I am not too well versed in Elixir/ShareDB.

[tmbb]

No, I haven’t had the time to work on this.

[tmbb]

It seems like Yjs is getting really capable, and even the author of ShareDB supports CRDTs inspired by Yjs as being the future of real time collaboration on the web. Has anyone looked into implementing am Elixir backend for Yjs?