FLAME failed with timeout in fly.io

I’m trying to setup flame on fly.io, but it failed with timeout errors. could you give me some idea if you have any?

environment:

Erlang/OTP 26 [erts-14.2.5.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit] [dtrace]
IEx 1.18.3 (compiled with Erlang/OTP 26)

{:flame, "~> 0.5.2"}

logs of flame worker node:

2025-04-25T15:11:36Z [info] INFO Preparing to run: `/app/bin/server` as nobody
2025-04-25T15:11:36Z [info] INFO [fly api proxy] listening at /.fly/api
2025-04-25T15:11:36Z [info]Machine created and started in 8.029s
2025-04-25T15:11:36Z [info]2025/04/25 15:11:36 INFO SSH listening listen_address=[fdaa:2:be18:a7b:3b0:c08b:f1fa:2]:22
2025-04-25T15:11:39Z [info]15:11:39.972 [info] starting with parent %FLAME.Parent{pid: #PID<85562.6154.0>, ref: #Reference<85562.3010329052.226492417.95334>, backend: FLAME.FlyBackend, node_base: "myapp-staging-159f7049-flame-5bcd1a05094c255608f0", flame_vsn: "0.5.2", backend_vsn: "0.5.2", backend_app: :flame, host_env: "FLY_PRIVATE_IP"}
2025-04-25T15:11:39Z [info]15:11:39.976 [info] connect (1) :"myapp-staging-159f7049-01JSPQHV6C4EKNCC73E9V5ZV1B@fdaa:2:be18:a7b:3ad:ede4:6e46:2": true
2025-04-25T15:11:40Z [info]15:11:40.157 [info] Discovered node :"myapp-staging-159f7049-01JSPQHV6C4EKNCC73E9V5ZV1B@fdaa:2:be18:a7b:3ad:ede4:6e46:2" in region fra
2025-04-25T15:11:40Z [info] WARN Reaped child process with pid: 699 and signal: SIGUSR1, core dumped? false
2025-04-25T15:12:09Z [info]15:12:09.979 [info] FLAME.Terminator.system_stop: parent pid #PID<85562.6154.0> went away :timeout. Going down
2025-04-25T15:12:10Z [info]15:12:10.005 [info] Postgrex.Protocol (#PID<0.5278.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.5282.0> exited
2025-04-25T15:12:10Z [info]15:12:10.020 [error] :gen_statem #PID<0.5278.0> terminating
2025-04-25T15:12:10Z [info]** (stop) exited in: GenServer.call(Postgrex.Parameters, {:insert, %{"DateStyle" => "ISO, MDY", "IntervalStyle" => "postgres", "TimeZone" => "UTC", "application_name" => "", "client_encoding" => "UTF8", "default_transaction_read_only" => "off", "in_hot_standby" => "off", "integer_datetimes" => "on", "is_superuser" => "on", "scram_iterations" => "4096", "server_encoding" => "UTF8", "server_version" => "16.6 (Ubuntu 16.6-1.pgdg24.04+1)", "session_authorization" => "postgres", "standard_conforming_strings" => "on"}}, 5000)

I’m generating test flame job with:

FLAME.call(MyApp.HTMLToPDFRunner, fn -> 123 end, timeout: :infinity)

at least it succeed in the LocalBackend (I mean, on my local).

application.ex

defmodule MyApp.Application do
  @moduledoc false

  use Application
  require Logger

  @impl true
  def start(_type, _args) do
    flame_instance? = not is_nil(FLAME.Parent.get())

    children =
      children_process_list(flame_instance?) ++
        Application.get_env(:myapp, :child_processes, [
          #...
        ])

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    events = [[:oban, :job, :start], [:oban, :job, :stop], [:oban, :job, :exception]]

    :telemetry.attach_many("oban-logger", events, &MyApp.ObanReporter.handle_event/4, [])
    Supervisor.start_link(children, opts)
  end

  # Process List for Flame Runner
  def children_process_list(true) do
    [
      #
    ]
  end

  def children_process_list(false) do
    [
      #...
      {FLAME.Pool,
       name: MyApp.HTMLToPDFRunner,
       min: 0,
       max: 10,
       max_concurrency: 10,
       idle_shutdown_after: :timer.minutes(3),
       boot_timeout: :timer.minutes(3),
       shutdown_timeout: :timer.minutes(3),
       log: :debug}
    ]
  end
end

runtime.ex

  config :flame, :backend, FLAME.FlyBackend

  config :flame, FLAME.FlyBackend,
    token: System.fetch_env!("FLY_API_TOKEN"),
    boot_timeout: 60_000,
    env: %{"RELEASE_COOKIE" => "***"}

  config :flame, :terminator, log: :info

somehow after I upgraded OTP from 26 to 27 it’s resolved. some incompatibility?