Fly, Phoenix Applications, Tailscale - how to cluster across platforms?

Hello!

I’ve been trying to get cross platform nodes talking to one another. As in:

1. Application on Fly
2. Application on AWS
3. Application on GCP

and having them all on one cluster over Tailscale. I had been following this link: Globally distributed Elixir over Tailscale · Richard Taylor for guidance.

Now, on a local separated network I can manually get the nodes connected using Node.connect(:some_tailscale_node). However, in a deployed state on Fly, it’s a bit more complicated.

First and foremost, this seems catered to Non-Elixir applications: Tailscale on Fly.io · Tailscale Docs. I may be wrong, but, it seems the stock Debian docker image doesn’t play well with Alpine compiled binaries. I could never get this multi-stage implementation to work.

Manually adding the dependencies on the runner image in my Dockerfile, I COULD get tailscaled up and running and then tailscale up --auth-key=etc would connect my Fly node to the mesh.

From there, I could start the server but trying to fly ssh console and get a remote terminal was no good. I’m assuming I’m not properly setting the network correctly in some way.

I’d like to do something like

export RELEASE_DISTRIBUTION=name
export RELEASE_NODE="test-app"@tailscale_ip

and then from another tailcale’d machine

Node.connect(:"test-app"@tailscale_ip)

Maybe I’m looking at this incorrectly, but I’d like to know if anyone else has investigated this issue and what their set up was like.

Perhaps there’s an easier way to cluster across platforms? I also looked at this: GitHub - fly-apps/tailscale-router · GitHub but haven’t dug too much into it yet.

For all of the above I’ve been using a standard, up to date Phoenix project using

mix phx.new hello_elixir --no-ecto

and deploying with the following Dockerfile

# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian
# instead of Alpine to avoid DNS resolution issues in production.
#
# https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=ubuntu
# https://hub.docker.com/_/ubuntu?tab=tags
#
# This file is based on these images:
#
#   - https://hub.docker.com/r/hexpm/elixir/tags - for the build image
#   - https://hub.docker.com/_/debian?tab=tags&page=1&name=bullseye-20240612-slim - for the release image
#   - https://pkgs.org/ - resource for finding needed packages
#   - Ex: hexpm/elixir:1.17.1-erlang-26.2.5-debian-bullseye-20240612-slim
#
ARG ELIXIR_VERSION=1.17.1
ARG OTP_VERSION=26.2.5
ARG DEBIAN_VERSION=bullseye-20240612-slim

ARG BUILDER_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

FROM ${BUILDER_IMAGE} as builder

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

# prepare build dir
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV="prod"

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config

# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile

COPY priv priv

COPY lib lib

COPY assets assets

# compile assets
RUN mix assets.deploy

# Compile the release
RUN mix compile

# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/

COPY rel rel
RUN mix release

# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE}

RUN apt-get update -y && \
  apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

WORKDIR "/app"
RUN chown nobody /app

# set runner ENV
ENV MIX_ENV="prod"

# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/hello_elixir ./

USER nobody

# If using an environment that doesn't automatically reap zombie processes, it is
# advised to add an init process such as tini via `apt-get install`
# above and adding an entrypoint. See https://github.com/krallin/tini for details
# ENTRYPOINT ["/tini", "--"]

CMD ["/app/bin/server"]

fly toml

app = 'host_name'
primary_region = 'atl'
kill_signal = 'SIGTERM'

[build]

[env]
  PHX_HOST = 'host_name'
  PORT = '8080'
  RELEASE_COOKIE = "test-cookie"

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[http_service.concurrency]
    type = 'connections'
    hard_limit = 1000
    soft_limit = 1000

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1