Crashes when Phoenix is starting up due to app dependency tree

Phoenix Forum Questions / Help
1

I recently updated phoenix to latest version 14.16.
While I start the application a socket is trying to reconnect to server. Since somehow the application is not yet fully started I get a lot of crashes like this:

[error] #PID<0.1221.0> running Carts.ApiWeb.Endpoint (connection #PID<0.1220.0>, stream id 1) terminated
Server: carts-api.zuppler.test:80 (http)
Request: GET /carts/socket/websocket?vsn=2.0.0
** (exit) an exception was raised:
    ** (ArgumentError) argument error
        (stdlib 3.12) :ets.lookup(Carts.ApiWeb.Endpoint, :code_reloader)
        lib/phoenix/endpoint.ex:542: Carts.ApiWeb.Endpoint.config/2
        (phoenix 1.4.16) lib/phoenix/socket/transport.ex:324: Phoenix.Socket.Transport.code_reload/3
        (phoenix 1.4.16) lib/phoenix/transports/websocket.ex:18: Phoenix.Transports.WebSocket.connect/4
        (phoenix 1.4.16) lib/phoenix/endpoint/cowboy2_handler.ex:21: Phoenix.Endpoint.Cowboy2Handler.init/4
        (cowboy 2.7.0) /Users/silviu/development/zuppler/carts-api/deps/cowboy/src/cowboy_handler.erl:41: :cowboy_handler.execute/2
        (cowboy 2.7.0) /Users/silviu/development/zuppler/carts-api/deps/cowboy/src/cowboy_stream_h.erl:320: :cowboy_stream_h.execute/3
        (cowboy 2.7.0) /Users/silviu/development/zuppler/carts-api/deps/cowboy/src/cowboy_stream_h.erl:302: :cowboy_stream_h.request_process/3
        (stdlib 3.12) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Since it happens continually the CPU goes to 100% and the errors keep flowing into out bugtracker.
I have not idea how to fix it. Any hits?

My guess is that somehow the phoenix app starts before :ets is initialized. Is this possible?

2

It is quite likely that, indeed, the problem is that you call :ets.lookup/2 before the ETS table is created. How do you create the table in your app?

3

I do not create any table myself. See the stack-trace, it comes from Phoenix.

4

Oh I see, sorry, I overlooked that.

In your config/dev.exs, do you have the following?

config :my_app, MyAppWeb.Endpoint,
  # ...
  code_reloader: true

If not, try to add it.

It seems that this code is checking that configuration, and if not found it defaults to endpoint.config(:code_reloader), that in turns checks that :ets table here.

5

Or, actually, it’s possible that the ETS table is not started because the endpoint is not started. Is the Carts.ApiWeb.Endpoint endpoint among the children in your application supervisor?

6

This is my config if helps:
application.ex:

defmodule Carts.Api.Application do
  use Application
  def start(_type, _args) do
    import Supervisor.Spec
    children = [
      supervisor(Endpoint, [])
    ]
    opts = [strategy: :one_for_one, name: Carts.Api.Supervisor]
    Supervisor.start_link(children, opts)
  end

  def config_change(changed, _new, removed) do
    Endpoint.config_change(changed, removed)
    :ok
  end
end

endpoint.ex

defmodule Carts.ApiWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :carts_api
  socket("/carts/socket", Carts.ApiWeb.CartSocket, websocket: true, longpoll: true, check_origin: false)

  plug(Plug.RequestId)
  plug(Plug.Logger)

  plug(
    Plug.Parsers,
    parsers: [:urlencoded, :multipart, :json],
    pass: ["*/*"],
    json_decoder: Jason
  )

  plug(Plug.MethodOverride)
  plug(Plug.Head)
  plug(
    Plug.Session,
    store: :cookie,
    key: "_carts_api_key",
    signing_salt: "xxx"
  )

  plug(CORSPlug, origin: [:self], headers: ["*"])

  plug(PipelineInstrumenter)
  plug(PrometheusExporter)

  plug(Plug.Static, at: "/", from: :carts_api, gzip: false, only: ~w(robots.txt favicon.ico))

  if code_reloading? do
    plug(Phoenix.LiveReloader)
    plug(Phoenix.CodeReloader)
  end

  plug(Carts.ApiWeb.Router)
end

config.exs:

use Mix.Config
config :carts_api, Carts.ApiWeb.Endpoint,
  url: [host: "localhost", port: 80],
  secret_key_base: "xxx",
  render_errors: [view: Carts.ApiWeb.ErrorView, accepts: ~w(html json)],
  pubsub: [name: Carts.ApiWeb.PubSub, adapter: Phoenix.PubSub.PG2],

config :phoenix, :json_library, Jason
config :phoenix, :stacktrace_depth, 40
import_config "#{Mix.env()}.exs"

dev.exs:

use Mix.Config

config :carts_api, Carts.ApiWeb.Endpoint,
  http: [port: 8881],
  url: [host: "carts-api.zuppler.test", port: 80],
  debug_errors: true,
  code_reloader: true,
  check_origin: false,
  server: false

config :carts_api, Carts.ApiWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r{priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$},
      ~r{priv/gettext/.*(po)$},
      ~r{lib/carts-api-web/controllers/.*(ex)$},
      ~r{lib/carts-api-web/channels/.*(ex)$},
      ~r{lib/carts-api-web/views/.*(ex)$},
      ~r{lib/carts-api-web/templates/.*(eex)$}
    ]
  ]
config :phoenix, :plug_init_mode, :runtime
7

It’s possible that there’s an extra requirement to add an app to those that must be started on bootup of your Phoenix app. Have you checked for something like that?

8

I went with your hint and I found out the issue (I think at least).
I am having multiple umbrella applications that start more phoenix endpoints. To delegate the requests to those apps I use master_proxy library.
Looking into logs I see that master proxy is started way ahead of my applications and it delegates the requests too soon.

[info] [master_proxy] Listening on http with options: [port: 8880, dispatch: [_: [{:_, MasterProxy.Cowboy2Handler, {nil, nil}}]]]
21:00:16.265 [info] Application master_proxy started on node nonode@nohost
.....
21:00:21.208 [info] Application carts_api started on node nonode@nohost

I am puzzled now. Is there any way to start a dependency after my applications has started?

9

One thing you might look into is this fork of master_proxy which turns it into a superviser which you can place in your application.exs. You might be able to better control the startup order that way but I’m not sure.

1 Like
10

This may help starting the application along with one of the umbrella apps but still there is no guarantee that all the other umbrella apps are started and this is the last one. I will dig more.

11

Another alternative to this fork is to use this solution. Append app: false to the deps and then start it manually.
But still I would have liked to be able to put master_proxy wait for all my phoenix applications to start first.

12

Create another application, which depends on all your ones with a phoenix endpoints. Start the proxy in that app‘s supervision tree.

13

Yes that is a solution but it seems like a hack to me. Still I will use it until I have a better option.

14

It’s not really a hack. Application dependencies are the determining factor of startup order between multiple beam applications. It your endpoints would be within a single application you could use the supervision tree, but as you’re using an umbrella/multiple applications adding additional applications is the way to go.

15

What I feel we lack here is a way to specify not also child dependencies, but also parent dependencies. Something like a list of parent_applications that need to be started after the current application. Like this we can control the dependency tree also above not just underneath.

16

In your case your proxy has a dependency on your multiple applications with those endpoints (like you wrote in different wording), while those phoenix applications can start just fine without the proxy (they’re just not accessable yet, but that’s not part of their application). The dependency is the other way round. What you’re proposing is imo way more of hack: It would probably be quite hard to resolve to a proper dependency tree by code as well as as for parsing it as a human. Imagine some of your dependencies using such an approach. I can’t imagine being able to infer the startup order from that.

2 Likes