Data Serialization in Phoenix Channels

our clients can have any backend stack to subscribe to our absinthe subscriptions, so i would say at least the most common ones: ruby, c#, java…

Can you elaborate on the tech stack you’re describing?

hey so if i can’t (because my client is a non-node-backend client) use the @absinthe/socket library, then i would need to implement all phoenix channels to be able to connect to absinthe subscriptions?

Okay thanks for the insights. I’ll hack away until I finish, lose interest, or get a more pressing consulting gig (this one is for a long-shot proposal, not for paying work).

This is your only option if you don’t want to integrate with the existing phoenix.js and/or absinthe socket js libraries.

Unfortunately it’s all or nothing as Phoenix channels is more than just a pipe on top of web sockets. There’s multiplexing, heartbeats, topics, request/response acknowledgements, etc. The reality is you need a full phoenix channels client to interact with Absinthe subscriptions. I don’t know what clojurescript’s js interop story is like but it seems like the path of least resistance would be to leverage the existing js libs. Of course it would be great to have a first class cljs channels client so if that’s the way you’d like to get then great!

Yes exactly! I am trying to push data down the websocket link with as little scaffolding as possible.
Which means I am trying to implement just enough of the Phoenix channels code on the clojurescript side to make the graphql subscriptions possible. I was hoping to just modify what is there in the re-graph code to play nicely with the Phoenix side, but I’m having a hard time discovering just what is needed for the client.

I have two branches of development. For this branch, I am not using phoenix.js (or rather, the absinthe socket JS code that uses phoenix.js under the hood to manage the sockets, etc). I already have the “regular” graphql queries working with the cljs re-graph library.

I have another branch that has ripped out the re-graph code and tries to use the absinthe socket library/apollo stuff. But in that case, I have to rewrite all the integration with re-frame that re-graph has already done. But the absinthe socket stuff annoys me because it seems like it was developed then has been stagnating. There is a dependency on core.js that is wrong (upgrade core-js · Issue #41 · absinthe-graphql/absinthe-socket · GitHub). I forked the project and fixed that (“core-js”: “^2”) and fixed some other issues with arrow functions, but really fixing that aging codebase properly requires reaching down into several libraries (“@jumpn/utils-array”, “@jumpn/utils-composite”, and “@jumpn/utils-graphql”), which haven’t been updated since 2018 or so.

There is work either way. I don’t relish fixing old code, so I was just hoping that there would be less work implementing the “extra” bits needed for the channels client.

Honestly the last time I tried this back in September or so, I went with the absinthe socket library, but gave up and went on to different projects before making any progress. This time I’m trying to stick with re-graph a little bit longer. From what I can tell, the graphql subscriptions don’t really use very much of the capabilities of Phoenix channels. But I’m also well aware that phoenix.js is quite complicated and handles lots of edge cases.

Maybe I should start by reimplementing phoenix.js in cljs, then use that to back re-graph.

It’s not clear what your goals are on the clojurescript side. Are you using a phoenix channels js client (ie phoenix.js) or are you trying to implement a channels client entirely on the clojurescript side? Merely shoving data on the wire on the websocdket connection won’t be enough, so it’s not clear what gaps you are trying to fill from the regraph => channel server. Since you are using Absinthe subscriptions, they have a javascript library which wraps phoenix.js: absinthe-socket/packages/socket at master · absinthe-graphql/absinthe-socket · GitHub

Is that not callable from clojurescript? I’m not at all familiar with regraph, but if your goals are to take client data from regraph and send it up to absinthe, I think the existing js has everything you need other than glueing the regraph object to the absinthe js client API?

Edit: I’m not whining…but re-reading this it reads like I am! I’m just trying to point out some difficulties I’m having navigating this problem, in the hopes that changes can be made that will smooth the path for someone else. I’m fine with working on this until I fix it, as it helps to learn the ins and outs of Phoenix and Elixir.

I’ve read the Phoenix.Socket doc you linked to. Nowhere in that doc does it define the array format expected by the JSON V2 serializer. It does link to docs on Phoenix.Socket.Serializer — Phoenix v1.4.13, which in turn says that JSON V2 is the default serializer, but here is no documentation of the two serializers, nor is there any instruction on how to insert my own serializer. There is only the message: “Custom serializers may be configured in the socket.”

I just grepped my project for V2 and Serializer etc etc, and I can’t see any existing config setting for that, nor do I see it in any of my code. It is like a secret handshake type of thing, which means it should be documented somewhere so that it isn’t so secret anymore. I can’t figure out how to change from V2 to V1 JSON serializer, let alone slot in my own.

I think what I need to do in this case is (A) write a custom Phoenix serializer to properly marshall and unmarshall the messages from re-graph, or (B) write a serializer inside of re-graph that talks JSON V1 or JSON V2. I have no idea how to do (A), so that leaves (B). To do (B), I’m looking for documentation of what the two serializers expect, which at this point is just read the source of the serializers. But the source code doesn’t describe the meaning behind the ref and join_ref fields.

In general, the docs for Phoenix are pretty great in all respects. I just think this is a gap that should be filled in some way.

There are a few things to unpack: %Phoenix.Socket.Message{} is the datatype used for messages within elixir. What’s send over the wire it actually fully up to the serializer used, while the question of “what is the wire” is solved by the channel transport.

Phoenix actually comes with two serializers: V1 uses json objects, while V2 uses arrays to save bytes (no keys); as well as two transports: long poll and websockets.

There’s already documentation about it here:

Sorry for resurrecting an old thread, but I’m doing this in place of opening an issue on GitHub.

Are the requirements for how to write a client for Phoenix sockets written down anywhere. I also saw another (even older) post on this forum here Specification of the protocol used by Phoenix channels - #10 by LostKobrakai.

My problem is that reading the docs here Phoenix.Socket.Message — Phoenix v1.8.8, it says:

The message format requires the following keys:

:topic - The string topic or topic:subtopic pair namespace, for example "messages", "messages:123"
:event- The string event name, for example "phx_join"
:payload - The message payload
:ref - The unique string ref
:join_ref - The unique string ref when joining

So reading that, it seems like a proper message should be a hash (JSON object) with keys. But this isn’t the case. As noted above, the serializers and the official javascript client both create an array, in a particular order:

let payload = [
      msg.join_ref, msg.ref, msg.topic, msg.event, msg.payload
    ]
    return callback(JSON.stringify(payload))

I came across this issue because I’m using Clojurescript on the client, and was having an impossible time trying to connect re-graph (a graphql client for the re-frame) to Phoenix/Absinthe subscriptions. Apparently the Lacinia server does something completely different, but it took me ages to drill down in the code enough to figure out that I need to send a JSON array, not a JSON object, and then to find exactly the order of the elements. I was so frustrated that I almost considered switching to a Clojure-based backend (but my Java error message PTSD pushed me to try harder to debug this!).

Given that there is no documentation of what is expected to communicate over this channel, I also don’t know the rules behind “ref” and “join_ref”. Can I just make up integers? Should “ref” increment with every new message sent over the channel? It certainly seems from the developer tools console that the Phoenix live code reloading heartbeat increments ref every time. The first message is

["3","3","phoenix:live_reload","phx_join",{}]

and then subsequent messages are:

[null,"4","phoenix","heartbeat",{}]
[null,"5","phoenix","heartbeat",{}]
...

Thanks in advance for pointers of what to read. I’m also willing to write a draft doc, something that could be linked in here: https://github.com/phoenixframework/phoenix/blob/master/guides/realtime/channels.md#client-libraries