Hi,

we’re building a collaborative learning platform in Meteor/React (FROG), where we’re using ShareDB heavily - both ot-json as the backing store for collaborative activities (activities are pluggable, and they get access to a shared document, which they can structure as they want - this gives us a lot of flexibility), and ot-text for collaborative text editing. We are having some problems scaling up (both because of Meteor and possibly ShareDB), and I dream of having a “OT as a service” thing, where I could just run an Elixir server or cluster, which would be compatible with the front-end ShareDB client (or something similar). All the JS code could be kept - and it would be super-fast, scaleable etc.

I did try a few years ago to rewrite the login in ot-text to Elixir, it was an interesting exercise… It’s pretty functional code (all functions which take an input and produce an output without side-effects), but they use closures a lot, which I changed to recursive functions. I even set up a way to run the (very extensive) JS test harness for ot-text against my library - and it got pretty far. I never fixed all the failing tests, and I kind of abandoned it, because I didn’t have the push to move it forwards (I was in the middle of running experiments for my PhD thesis, and didn’t actually need it. My very old abandoned attempt.

I still think OT is amazingly powerful, and it seems tailor-made for Elixir… I would love to see someone try again to make a production-ready OT server in Elixir - having a compatibility mode with ShareDB would let a lot of products switch easily, and would work well with existing extensions like Rich text etc, but investigating a more efficient transmission format would be interesting too…

Note that the ShareDB people have been working on a new version of ot-json for years, which apparently is soon ready.

If you pursue this, please let me know, as I’d be very interested to follow along, and maybe use it. We’re also working on collaborative writing analytics (predict collaboration quality etc), if anyone is interested.

best
Stian Håklev
Ecole Polytechnique Fédérale de Lausanne

Back from the dead. As part of the plan of writing a ShareDB backend in Elixir (so that we get all the real-time collaboration features for free), I’ve had to solve the problem of protocol mismatch between Phoenix and the ShareDB client. The ShareDB client is supposed to work with raw websockets, while Phoenix can only work in Channels. You can’t use raw websockets with Phoenix without getting yourself in a world of pain.

The obvious solution is to implement a websocket on top of Phoenix channels. If this makes as little sense as it made to me at the time, read on. The trick is to write a Javascript object that implements the websocket API but sends and receives messages over a Phoenix Channel.

I’ve ended up with something like this: main.js · GitHub

As you can see there, I build a PhoenixWS, which is an object that talks like a websocket but sends and receives messages over a Phoenix channel:

const connection = new PhoenixWS(socket, "room:lobby", {});

It takes as arguments a phoenix socket, a channel topic and an initial payload (which is useful for authentication, for example). This is the only phoenix-specific part of that file. From that point on, the javascript code is written as if it were a normal websocket. The rest of the code in that file was actually copied with no changes from a random Javascript “chat server” implementation, meant to work with a websocket on top of NodeJS. This chat server now works with a Phoenix backend (see here: GitHub - tmbb/phwocket_example: An example using PhoenixWS · GitHub)

The implementation is not complete and this is not production-ready yet, but it shows how easy it is to achieve some basic compatibility with raw websockets while being able to use Phoenix channels underneath.

The Elixir side is actually quite easy too. You add your PhoenixWS.Channel to your socket:

defmodule PhwocketExampleWeb.UserSocket do
  use Phoenix.Socket
  require PhoenixWS.Socket

  ## Channels (actually a very weird channel that talks like a websocket)
  PhoenixWS.Socket.channel("room:*", PhwocketExampleWeb.RoomWSChannel)
  # ...
end

The code that implements the websocket channel is actually pretty easy to write:

defmodule PhwocketExampleWeb.RoomWSChannel do
  # Import some convenience macros to abstract away the implementation details
  use PhoenixWS.Channel, web: PhwocketExampleWeb

  def join("room:" <> _, _payload, socket) do
    {:ok, socket}
  end

  # Handle messages from the Client
  def phoenix_ws_in(data, socket) do
    # This is just an echo server, so we just broadcast the same message back into the client
    broadcast!(socket, data)
    {:noreply, socket}
  end
end

Although there are still some holes in the websocket implementation (it doesn’t close properly, for instance), this can serve as a bridge between the ShareDB client and a backend that runs behind Phoenix. I imagine that the PhoenixWS package might be useful by itself in projects where you want to interface directly with a Javascript library that uses websockets.

Expect more developments soon, but not thaaat soon.

After a long time, I’ve decided to go back to realtime collaboration on Phoenix. The plan is to implement something compatible with ShareDB’s protocol. There are already some tools using ShareDB in the wild, and the most important case (collaborative editing of text) is already covered by ´ShareDB` pretty well (there’s a demo under 20 LOC doing exactly that with some extensions).

ShareDB has a client and a server implementation, both written in Javascript. In an ideal world, I’d port the protocol to Elixir and reuse the client without requiring any changes. However, the protocol expects messages to be sent through “bare” websockets, and Phoenix expects messages to be sent through “channels”. Channels are ingrained so deeply in Phoenix that I don’t think it’s possible to use bare websockets wthout reimplementing the whole channel machinery. In fact, Phoenix doesn’t even know what an websocket is.

The client-side API for ShareDB starts with something like this:

var ShareDB = require('sharedb/lib/client');
var socket = new WebSocket('ws://localhost:8080');
var connection = new ShareDB.Connection(socket);

The socket really must be a WebSocket, or at least something that implements the websocket methods. That’s actually very easy with Phoenix channels: you just have to create an object (class, prototype, whatever) which implements those methods and behaves as a webocket. However, there is a problem.

ShareDB allows you to create several “documents”, which can be updated in parallel. That means each document should live in its own process and be connected to the client though its own topic. Our “custom” websocket implementation would have to route each message into the appropriate phoenix topic. But the WebSocket object is sent raw JSON and doesn’t know which document/topic the message refers to. The only way to know that is to parse the JSON again and extract the document’s ID, which is unacceptable.

This means I will have to rewrite both the Connection object and the WebSocket object so that the WebSocket is compatible with Phoenix channels and the Connection object can send objects into the WebSocket (so that it can route the messages into the correct topics).