Can't capture a livebook "up" in a cluster

I am experimenting with how to capture a Livebook connection in a cluster. I run clustered containers: 2 Phoenix nodes and a Livebook to monitor the web app. I can’t capture the Livebook “nodeup” (or down) event in the web apps.

I use libcluster in Gossip mode. It is set in the web app and in the Livebook (see below). The 3 nodes are clustered: I have connection logs from libcluster and I can :rpc.call a Phoenix node from the LB.

However, libcluster sends only logs: [info] [libcluster:gossip] connected to... but no event.

I use :net_kernel.monitor_nodes(true) in the web app but this only captures the Phoenix nodes events, not the Livebook node.

I tried to use Phoenix.PubSub with the registered MyApp.PubSub in the Livebook to push a message once connected: I have a subscription in a module of the web app to a topic, set up a “catch all” handle_info on this topic. However, the broadcasted message from the Livebook is not received?/captured? in the web app. Is this a config (naming) problem?

The Livebook image (GitHub - livebook-dev/livebook: Automate code & data workflows with interactive Elixir notebooks) contains:

Mix.install ([
{:libcluster, "~> 3.3.3"},
{:phoenix_pubsub, "~> 2.1"}
])

topologies = [gossip: [strategy: Cluster.Strategy.Gossip]]
opts = [strategy: :one_for_one, name: MyApp.Supervisor]
children = [
  {Phoenix.PubSub, name: MyApp.PubSub},
  {Cluster.Supervisor, [topologies, [name: MyApp.ClusterSupervisor]]}
]
Supervisor.start_link(children, opts)

Phoenix.PubSub.broadcast(MyApp.PubSub, "nodes", {:up, node()})

In a GenServer module in the web app:

def init(_) do
  :net_kernel.monitor_nodes(true)
  Phoenix.PubSub.subscribe(MyApp.PubSub, "nodes")
{:ok, nil}

def handle_info({:nodeup, node}, _state) do
  # this is ok between web apps
{:noreply, nil}

def handle_info(msg, _state) do
 # !! no message on the "nodes" topic is captured here
{:noreply, nil}

The docker-compose service for the Livebook is:

livebook:
    image: ghcr.io/livebook-dev/livebook
    networks:
      - mynet
    environment:
      # - MIX_ENV=prod
      - LIVEBOOK_DISTRIBUTION=name
      - LIVEBOOK_COOKIE=supersecret
      - LIVEBOOK_PASSWORD=securesecret
      - LIVEBOOK_NODE=livebook
    hostname: livebook.mynet
    volumes:
      - ./data:/data/
      - ./db:/db
    ports:
      - "8080:8080"
      - "8081:8081"
1 Like

This is why we need to be able to embed livebook in a normal phoenix router. Coming soon…

2 Likes

@ndrean Livebook spawns a separate node for every notebook and those nodes are hidden (so that notebooks are not clustered with each other by default). Try :net_kernel.monitor_nodes(true, %{node_type: :all}) and see if you get :nodeup.

2 Likes

Indeed, this hidden info worked! Thks @jonatanklosko

{:nodeup, :"rf6qapyv-livebook_server@livebook", %{node_type: :hidden}}
[libcluster:gossip] connected to :"rf6qapyv-livebook_server@livebook"

There is still one point I don’t understand. I can Phoenix.PubSub.direct_broadcast from the Livebook to a particular node and receive it (in a Channel of the node who subscribed) but a simple broadcast isn’t received.

but a simple broadcast isn’t received.

By default Phoenix.PubSub uses :pg2 underneath and hidden nodes are not visible as group members on the other nodes.

@ityonemo Do you mean like the LiveDashboard? on the same port?

1 Like

Wouldn’t that be nice :grimacing: