Deployed to fly.io - works locally seems to deploy OK with postgres db etc. but crashes on start

Phoenix Forum Questions / Help
, ,
1 
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]=SUPERVISOR REPORT==== 20-May-2024::08:55:53.170641 ===
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    supervisor: {local,kernel_sup}
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    errorContext: start_error
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    reason: {on_load_function_failed,gl}
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    offender: [{pid,undefined},
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]               {id,kernel_safe_sup},
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]               {mfargs,{supervisor,start_link,
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]                                   [{local,kernel_safe_sup},kernel,safe]}},
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]               {restart_type,permanent},
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]               {significant,false},
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]               {shutdown,infinity},
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]               {child_type,supervisor}]
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]=CRASH REPORT==== 20-May-2024::08:55:53.177485 ===
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]  crasher:
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    initial call: supervisor:kernel/1
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    pid: <0.3553.0>
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    registered_name: []
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    exception exit: {on_load_function_failed,gl}
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]      in function  init:run_on_load_handlers/0
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]      in call from kernel:init/1 (kernel.erl, line 189)
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]      in call from supervisor:init/1 (supervisor.erl, line 330)
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]      in call from gen_server:init_it/2 (gen_server.erl, line 423)
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]      in call from gen_server:init_it/6 (gen_server.erl, line 390)
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    ancestors: [kernel_sup,<0.3527.0>]
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    message_queue_len: 0
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    messages: []
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    links: [<0.3529.0>]
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    dictionary: []
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    trap_exit: true
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    status: running
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    heap_size: 376
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    stack_size: 29
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]    reductions: 228
2024-05-20T08:55:53Z app[e7843ee2b36368] lhr [info]  neighbours:
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info] WARN Reaped child process with pid: 371 and signal: SIGUSR1, core dumped? false
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]=CRASH REPORT==== 20-May-2024::08:55:54.180210 ===
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]  crasher:
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    initial call: application_master:init/4
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    pid: <0.3526.0>
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    registered_name: []
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    exception exit: {{shutdown,
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]                         {failed_to_start_child,kernel_safe_sup,
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]                             {on_load_function_failed,gl}}},
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]                     {kernel,start,[normal,[]]}}
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]      in function  application_master:init/4 (application_master.erl, line 142)
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    ancestors: [<0.3525.0>]
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    message_queue_len: 1
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    messages: [{'EXIT',<0.3527.0>,normal}]
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    links: [<0.3525.0>,<0.3524.0>]
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    dictionary: []
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    trap_exit: true
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    status: running
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    heap_size: 610
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    stack_size: 29
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    reductions: 166
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]  neighbours:
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]=INFO REPORT==== 20-May-2024::08:55:54.185443 ===
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    application: kernel
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    exited: {{shutdown,
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]                 {failed_to_start_child,kernel_safe_sup,
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]                     {on_load_function_failed,gl}}},
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]             {kernel,start,[normal,[]]}}
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]    type: permanent
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_funKernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,gl}}},{kernel,start,[normal,[]]}}})
2024-05-20T08:55:54Z app[e7843ee2b36368] lhr [info]Crash dump is being written to: erl_crash.dump...ction_failed,gl}}},{kernel,start,[normal,[]]}}}"}
2024-05-20T08:55:54Z proxy[178151d2b4e268] lhr [info]Starting machine
2024-05-20T08:55:54Z proxy[178151d2b4e268] lhr [error][PM07] failed to change machine state: machine still active, refusing to start
2

fly.toml

# fly.toml app configuration file generated for songsheet on 2024-05-18T07:16:55+01:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = 'songsheet'
primary_region = 'lhr'
kill_signal = 'SIGTERM'

[build]

[deploy]
  release_command = '/app/bin/migrate'

[env]
  PHX_HOST = 'songsheet.fly.dev'
  PORT = '8080'

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

  [http_service.concurrency]
    type = 'connections'
    hard_limit = 1000
    soft_limit = 1000

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1

Dockerfile

# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian
# instead of Alpine to avoid DNS resolution issues in production.
#
# https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=ubuntu
# https://hub.docker.com/_/ubuntu?tab=tags
#
# This file is based on these images:
#
#   - https://hub.docker.com/r/hexpm/elixir/tags - for the build image
#   - https://hub.docker.com/_/debian?tab=tags&page=1&name=bullseye-20240408-slim - for the release image
#   - https://pkgs.org/ - resource for finding needed packages
#   - Ex: hexpm/elixir:1.16.2-erlang-26.2.3-debian-bullseye-20240408-slim
#
ARG ELIXIR_VERSION=1.16.2
ARG OTP_VERSION=24.0.6
ARG DEBIAN_VERSION=buster-20240513-slim

ARG BUILDER_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

FROM ${BUILDER_IMAGE} as builder

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

# prepare build dir
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV="prod"

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config

# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile

COPY priv priv

COPY lib lib

COPY assets assets

# compile assets
RUN mix assets.deploy

# Compile the release
RUN mix compile

# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/

COPY rel rel
RUN mix release

# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE}

RUN apt-get update -y && \
  apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

WORKDIR "/app"
RUN chown nobody /app

# set runner ENV
ENV MIX_ENV="prod"

# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/songsheet ./

USER nobody

# If using an environment that doesn't automatically reap zombie processes, it is
# advised to add an init process such as tini via `apt-get install`
# above and adding an entrypoint. See https://github.com/krallin/tini for details
# ENTRYPOINT ["/tini", "--"]

CMD ["/app/bin/server"]

runtime.exs

import Config

# config/runtime.exs is executed for all environments, including
# during releases. It is executed after compilation and before the
# system starts, so it is typically used to load production configuration
# and secrets from environment variables or elsewhere. Do not define
# any compile-time configuration in here, as it won't be applied.
# The block below contains prod specific runtime configuration.

# ## Using releases
#
# If you use `mix release`, you need to explicitly enable the server
# by passing the PHX_SERVER=true when you start it:
#
#     PHX_SERVER=true bin/songsheet start
#
# Alternatively, you can use `mix phx.gen.release` to generate a `bin/server`
# script that automatically sets the env var above.
if System.get_env("PHX_SERVER") do
  config :songsheet, SongsheetWeb.Endpoint, server: true
end

if config_env() == :prod do
  database_url =
    System.get_env("DATABASE_URL") ||
      raise """
      environment variable DATABASE_URL is missing.
      For example: ecto://USER:PASS@HOST/DATABASE
      """

  maybe_ipv6 = if System.get_env("ECTO_IPV6") in ~w(true 1), do: [:inet6], else: []

  config :songsheet, Songsheet.Repo,
    # ssl: true,
    url: database_url,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10"),
    socket_options: maybe_ipv6

  # The secret key base is used to sign/encrypt cookies and other secrets.
  # A default value is used in config/dev.exs and config/test.exs but you
  # want to use a different value for prod and you most likely don't want
  # to check this value into version control, so we use an environment
  # variable instead.
  secret_key_base =
    System.get_env("SECRET_KEY_BASE") ||
      raise """
      environment variable SECRET_KEY_BASE is missing.
      You can generate one by calling: mix phx.gen.secret
      """

  host = System.get_env("PHX_HOST") || "example.com"
  port = String.to_integer(System.get_env("PORT") || "8080")

  config :songsheet, :dns_cluster_query, System.get_env("DNS_CLUSTER_QUERY")

  config :songsheet, SongsheetWeb.Endpoint,
    url: [host: host, port: 8080, scheme: "https"],
    http: [
      # Enable IPv6 and bind on all interfaces.
      # Set it to  {0, 0, 0, 0, 0, 0, 0, 1} for local network only access.
      # See the documentation on https://hexdocs.pm/bandit/Bandit.html#t:options/0
      # for details about using IPv6 vs IPv4 and loopback vs public addresses.
      ip: {0, 0, 0, 0, 0, 0, 0, 0},
      port: port
    ],
    secret_key_base: secret_key_base

  # ## SSL Support
  #
  # To get SSL working, you will need to add the `https` key
  # to your endpoint configuration:
  #
  #     config :songsheet, SongsheetWeb.Endpoint,
  #       https: [
  #         ...,
  #         port: 443,
  #         cipher_suite: :strong,
  #         keyfile: System.get_env("SOME_APP_SSL_KEY_PATH"),
  #         certfile: System.get_env("SOME_APP_SSL_CERT_PATH")
  #       ]
  #
  # The `cipher_suite` is set to `:strong` to support only the
  # latest and more secure SSL ciphers. This means old browsers
  # and clients may not be supported. You can set it to
  # `:compatible` for wider support.
  #
  # `:keyfile` and `:certfile` expect an absolute path to the key
  # and cert in disk or a relative path inside priv, for example
  # "priv/ssl/server.key". For all supported SSL configuration
  # options, see https://hexdocs.pm/plug/Plug.SSL.html#configure/1
  #
  # We also recommend setting `force_ssl` in your config/prod.exs,
  # ensuring no data is ever sent via http, always redirecting to https:
  #
  #     config :songsheet, SongsheetWeb.Endpoint,
  #       force_ssl: [hsts: true]
  #
  # Check `Plug.SSL` for all available options in `force_ssl`.

  # ## Configuring the mailer
  #
  # In production you need to configure the mailer to use a different adapter.
  # Also, you may need to configure the Swoosh API client of your choice if you
  # are not using SMTP. Here is an example of the configuration:
  #
  #     config :songsheet, Songsheet.Mailer,
  #       adapter: Swoosh.Adapters.Mailgun,
  #       api_key: System.get_env("MAILGUN_API_KEY"),
  #       domain: System.get_env("MAILGUN_DOMAIN")
  #
  # For this example you need include a HTTP client required by Swoosh API client.
  # Swoosh supports Hackney and Finch out of the box:
  #
  #     config :swoosh, :api_client, Swoosh.ApiClient.Hackney
  #
  # See https://hexdocs.pm/swoosh/Swoosh.html#module-installation for details.
end

I haven’t figured out what the problem is :frowning:

Could this be wrong version of erlang/elixir/phoenix?

elixir
 *1.16.2-otp-26
erlang
 *26.2.3

    {:phoenix, "~> 1.7.11"},
      {:phoenix_ecto, "~> 4.4"},
      {:ecto_sql, "~> 3.10"},
      {:postgrex, ">= 0.0.0"},
      {:phoenix_html, "~> 4.0"},
      {:phoenix_live_reload, "~> 1.2", only: :dev},
      {:phoenix_live_view, "~> 0.20.2"},
      {:floki, ">= 0.30.0", only: :test},
      {:phoenix_live_dashboard, "~> 0.8.3"},
      {:esbuild, "~> 0.8", runtime: Mix.env() == :dev},
      {:tailwind, "~> 0.2", runtime: Mix.env() == :dev},
      {:heroicons,
       github: "tailwindlabs/heroicons",
       tag: "v2.1.1",
       sparse: "optimized",
       app: false,
       compile: false,
       depth: 1},
      {:swoosh, "~> 1.5"},
      {:finch, "~> 0.13"},
      {:telemetry_metrics, "~> 0.6"},
      {:telemetry_poller, "~> 1.0"},
      {:gettext, "~> 0.20"},
      {:jason, "~> 1.2"},
      {:dns_cluster, "~> 0.1.1"},
      {:ash, "~> 3.0.0"},
      {:ash_postgres, "~> 2.0.0"},
      {:credo, "~> 1.7.5"},
      {:bandit, "~> 1.2"}

I am running latest on my machine

Thanks! :smiley:

2

There’s a large OTP version mismatch between the docker build and what I assume is your local asdf installation, but elixir 1.16 supports erlang 24 so it can’t be that… perhaps another dependency is having issues?

Have you tried to build a release locally with that version of erlang to confirm “it works on my machine”? Don’t forget to also asdf install elixir 1.16.2-otp-24.

1 Like
3

I think LostKobrakai said in another thread lately that OTP 24 is not supported in Elixir 1.16?

But even if it is (and I think it should be) I still agree with you that the version mismatch is too big.

4

Understood, I found it hard to tell from the log.

I will amend and try again.

Thank you both

5

It does: Compatibility and deprecations — Elixir v1.16.2

6

Then sorry, I remembered wrongly.

7

Looks like this error is related to wx. Check this post: Update to Erlang 24 causing a startup crash - Failed to load NIF library - #3 by akagr

If you don’t have wx anywhere listed in your local applications, I would recommend checking your dependencies, even though the ones you listed shouldn’t have wx as their application.

1 Like
8

On a slightly unrelated note, this setting is for generating url links with the endpoint helper functions, so port should be 443 otherwise your public-facing urls will be generated with :8080.

1 Like
9

Appreciate the extra effort here Juan!

Thank you

1 Like