Oban bringing app down with `Repo not available` error In development

jozuas · October 27, 2025, 6:41pm

@josevalim says it’s a bug Larger xref module cycles after upgrading from 1.18 -> 1.19 · Issue #14844 · elixir-lang/elixir · GitHub, but the problem is that I’m not managing to create some minimal example for replication. Maybe one of you (@nicanor, @stevegrossi) that are having this problem on Elixir 1.19 have a public project or some small example that could replicate this?

josevalim · October 27, 2025, 9:06pm

If you folks have a stacktrace, can you please share it?

Also, does it happen in both OTP 27 and 28?

stevegrossi · October 28, 2025, 7:41pm

Sure thing, I’ve confirmed it happens on both elixir 1.19.1-otp-28/erlang 28.1.1 as well as elixir 1.19.1-otp-27/erlang 27.3.4.4

This is the stacktrace (actual application name replaced with my_app):

Server: dev.localhost:4000 (http)
Request: GET /campaigns
** (exit) an exception was raised:
    ** (RuntimeError) could not lookup Ecto repo MyApp.Repo because it was not started or it does not exist
        (ecto 3.13.4) lib/ecto/repo/registry.ex:22: Ecto.Repo.Registry.lookup/1
        (ecto 3.13.4) lib/ecto/repo/supervisor.ex:176: Ecto.Repo.Supervisor.tuplet/2
        (my_app 0.0.1) lib/my_app/repo.ex:4: MyApp.Repo.get_by/3
        (my_app 0.0.1) lib/my_app/teams/teams.ex:697: MyApp.Teams.load_team_from_subdomain/1
        (my_app 0.0.1) lib/my_app_web/plugs/load_team_from_subdomain.ex:16: MyAppWeb.Plugs.LoadTeamFromSubdomain.call/2
        (my_app 0.0.1) MyAppWeb.Router.with_team/2
        (my_app 0.0.1) lib/my_app_web/router.ex:1: MyAppWeb.Router.__pipe_through4__/1
        (phoenix 1.8.1) lib/phoenix/router.ex:407: Phoenix.Router.__call__/5
        (my_app 0.0.1) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.plug_builder_call/2
        (my_app 0.0.1) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint."call (overridable 3)"/2
        (my_app 0.0.1) deps/plug/lib/plug/debugger.ex:155: MyAppWeb.Endpoint."call (overridable 4)"/2
        (my_app 0.0.1) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.call/2
        (phoenix 1.8.1) lib/phoenix/endpoint/sync_code_reload_plug.ex:22: Phoenix.Endpoint.SyncCodeReloadPlug.do_call/4
        (plug_cowboy 2.7.4) lib/plug/cowboy/handler.ex:11: Plug.Cowboy.Handler.init/2
        (cowboy 2.14.2) /Users/steve/Work/monorepo/my_app/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2
        (cowboy 2.14.2) /Users/steve/Work/monorepo/my_app/deps/cowboy/src/cowboy_stream_h.erl:310: :cowboy_stream_h.execute/3
        (cowboy 2.14.2) /Users/steve/Work/monorepo/my_app/deps/cowboy/src/cowboy_stream_h.erl:299: :cowboy_stream_h.request_process/3
        (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3

file=application_controller.erl line=2129 [notice] Application my_app exited: shutdown

jozuas · October 28, 2025, 8:44pm

Ooh, I tried over and over again, but I cannot replicate with OTP27. OTP27 still shows that there’s a rebuild after iex start, but it does not result in 700+ files being recompiled unlike with OTP28:

iex --name "tp@127.0.0.1" --cookie dev -S mix run
Erlang/OTP 27 [erts-15.2.7.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

20:36:43.734 pid=<0.274.0> module=Appsignal function=initialize/0 line=89  [info] AppSignal disabled.
20:36:44.181 pid=<0.649.0> module=Bandit function=start_link/1 line=321  [info] Running TpWeb.Endpoint with Bandit 1.8.0 at 0.0.0.0:4000 (http)
20:36:44.185 pid=<0.471.0> module=Phoenix.Endpoint.Supervisor function=log_access_url/2 line=435  [info] Access TpWeb.Endpoint at http://localhost:4000
Interactive Elixir (1.19.0) - press Ctrl+C to exit (type h() ENTER for help)

  ../priv/static/assets/app.js  5.0mb ⚠️

⚡ Done in 107ms

Rebuilding...
20:36:44.637 pid=<0.1120.0> module=Phoenix.LiveReloader.Channel function=handle_info/2 line=48  [debug] Live reload: priv/static/assets/app.js

Done in 786ms.
20:36:45.755 pid=<0.1139.0> module=Phoenix.Logger function=phoenix_router_dispatch_start/4 line=284  [debug] Processing with TpWeb.PageController.landing_page/2
  Parameters: %{}
  Pipelines: [:browser, :public_layout]
iex(tp@127.0.0.1)1>

Elixir 1.19 OTP28 stacktrace:

iex --name "tp@127.0.0.1" --cookie dev -S mix run
Erlang/OTP 28 [erts-16.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

20:40:51.180 pid=<0.274.0> module=Appsignal function=initialize/0 line=89  [info] AppSignal disabled.
20:40:51.631 pid=<0.649.0> module=Bandit function=start_link/1 line=321  [info] Running TpWeb.Endpoint with Bandit 1.8.0 at 0.0.0.0:4000 (http)
20:40:51.636 pid=<0.471.0> module=Phoenix.Endpoint.Supervisor function=log_access_url/2 line=435  [info] Access TpWeb.Endpoint at http://localhost:4000
Interactive Elixir (1.19.0) - press Ctrl+C to exit (type h() ENTER for help)

  ../priv/static/assets/app.js  5.0mb ⚠️

⚡ Done in 108ms

Rebuilding...
Compiling 757 files (.ex)
20:22:50.233 pid=<0.1019.0> module=gen_server function=error_info/7 line=2785  \[error\] GenServer {Oban.Registry, {Oban, Oban.Met.Examiner}} terminating
\*\* (UndefinedFunctionError) function TpData.Repo.query/3 is undefined (module TpData.Repo is not available)
(tp 1.0.0) TpData.Repo.query(“SELECT pg_notify($1, payload) FROM json_array_elements_text($2::json) AS payload”, \[“public.oban_gossip”, \[“H4sIAAAAAAAAE72Yy27cOBBF/0Vr0+D74dX8wWxmlSAQimTRJiJRPRI1iRH434cNI8gANhBZLQ16RXWpSR3currVP7rwhOHr0j18/tEVGLF76P70ULq7rkzxuqqXPxg397R9WLs6r6Xk8tjqv9x1f6+4XmuW56Xi2Jep5pQD1DyVpdUuFeaKsYfaajjlijBKuP2L0wfOH6S7l0Y7KT610mEKMPRDHnOrZW0bqPhzWdZhuOseh8n/Knm9tq45tp+mzAEPJpAobCDGcUc8pUAkKBoiYEyo2h4XWBds9QmGBdvNlwi/P528nm55WmucvpX+v090PcLL3Q3U4hTWEUvtF4Q5PG0FptTRwLRIxMkoCAMZk4BogtE7gb2e7hxgKQ/Y16mv+L1upeXYobQ8MV4hsc4pgiAFFZ6y5OlOWo6fR2t5LqFfVr+EOV+uLblZYG87kt8mMOoZsUw54gR3QQnrwPm9AjuxI1sz5vrcj9DsbO79moeI81Zs1h3dl8kk4pmXJEjUXictRIh7lUZPNDJMsA6bW1LLt6DobaR460kXQyBSJhMEDZ4Z2ElKn2v53ze/GPnhmLQEAsAN4SYp5qQW2u61LqnPw+R5P+bH+WNBwtLDeQnFibNBEBDGRkNl0GIvL3tiAxb8thmTOV5WXgtibUtdQSrPIAYIYm9+MOI8TDMmnLEE7JdpnQNuZqaPDRENmWFNWShZi1xMCmmTRbfbsE4MEX4qsb/kCw654GaJvWPwNwYIpYh3vi2xpXll5PXrvQo70d/rDPHD6mLvGNdtuJj3rSGbeXmjODPoHci9ExA/0bfGteQPq8sePv8khwQ4R0JFmxTRaU+F2GvzJ84/YcaYa59LxWGzdZk3tNSN4mqjj0PVLD+BSoY2u6d7w7y2J4b5cYTNU6I+/KUY0BIP0pJkVIpJUrAq7A3v5n/qwd5D3f5PxDuJS9xIzURFILbttBFWeOZQp71Gb9mZI8+Q/8EZ/PCBFHG4z8toiFUWiefcGZvakOj2Gpf+jc9/efkXo4gkwCwUAAA=”\]\], \[prefix: “public”, log: false, oban: true, telemetry_options: \[oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Pro.Engines.Smart, get_dynamic_repo: nil, insert_trigger: true, log: false, name: Oban, node: “tp@127.0.0.1”, notifier: {Oban.Notifiers.Postgres, [ ]}, peer: {Oban.Peers.Database, [ ]}, plugins: \[{Oban.Pro.Plugins.DynamicQueues, \[sync_mode: :automatic, queues: \[default: \[limit: 10\], file_to_text: \[limit: 1\], document_search: \[limit: 1\], docx: \[limit: 10\], credit_intel: \[limit: 5\]\]\]}\], prefix: “public”, queues: [ ], repo: TpData.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}\]\])
(oban 2.20.1) lib/oban/notifiers/postgres.ex:101: Oban.Notifiers.Postgres.notify/3
(oban 2.20.1) lib/oban/notifier.ex:204: anonymous fn/4 in Oban.Notifier.notify/3
(telemetry 1.3.0) /Users/juozas/dev/terrapin-app/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
(oban_met 1.0.3) lib/oban/met/examiner.ex:115: Oban.Met.Examiner.handle_info/2
(stdlib 7.1) gen_server.erl:2434: :gen_server.try_handle_info/3
(stdlib 7.1) gen_server.erl:2420: :gen_server.handle_msg/3
(stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Last message: :check

josevalim · October 28, 2025, 10:00pm

@jozuas What is inside def application in your mix.exs?

@stevegrossi try running your app with elixir --logger-sasl-reports true -S mix phx.server. It will include supervion reports and it should show if apps or supervision trees are shutting down (which would explain the missing repo).

jozuas · October 29, 2025, 1:23pm

Sure, here it is:

  def application do
    [
      extra_applications: extra_applications(Mix.env()),
      mod: {Tp.Application, []}
    ]
  end

  defp extra_applications(:dev) do
    [
      :logger,
      :xmerl,
      :wx,
      :observer,
      :runtime_tools,
      :tools
    ]
  end

  defp extra_applications(_env) do
    [
      :logger,
      :xmerl,
      :runtime_tools
    ]
  end

josevalim · October 29, 2025, 2:46pm

Thank you. Can you run the v1.19 branch from Elixir and see if you spot anything different? I have a theory this is related to regexes in .app files, but the issue is certainly not coming from your .app file.

vkryukov · October 29, 2025, 3:19pm

Not sure if it’s related, but there was a thread recently with similar symptoms where kino dependency was found to be a culprit: Recompilation triggered after request in dev even without code changes (iex -S mix phx.server) - #9 by axelson

jozuas · October 29, 2025, 4:42pm

I cannot reproduce the stacktrace after running my app with this commit in elixir 1.19 branch. However, now each invocation of iex goes through the same analyzing / compiling steps even though there are no code changes:

iex --name "tp@127.0.0.1" --cookie dev -S mix run
Erlang/OTP 28 [erts-16.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

mix deps.compile running across 6 OS processes
5> ===> Analyzing applications...
4> ===> Analyzing applications...
3> ===> Analyzing applications...
6> ===> Analyzing applications...
1> ===> Analyzing applications...
2> ===> Analyzing applications...
5> ===> Compiling telemetry
4> ===> Compiling mimerl
1> ===> Compiling sleeplocks
3> ===> Compiling erlsom
6> ===> Compiling mochiweb
2> ===> Compiling unicode_util_compat
3> ===> Analyzing applications...
5> ===> Analyzing applications...
4> ===> Analyzing applications...
1> ===> Analyzing applications...
2> ===> Analyzing applications...
4> ===> Compiling certifi
5> ===> Compiling telemetry_poller
3> ===> Compiling metrics
1> ===> Compiling parse_trans
2> ===> Compiling idna
2> ===> Analyzing applications...
2> ===> Compiling hackney
Compiling 758 files (.ex)
Generated tp app
16:36:24.148 pid=<0.5020.0> module=Appsignal function=initialize/0 line=89  [info] AppSignal disabled.
16:36:24.691 pid=<0.5395.0> module=Bandit function=start_link/1 line=321  [info] Running TpWeb.Endpoint with Bandit 1.8.0 at 0.0.0.0:4000 (http)
16:36:24.699 pid=<0.5217.0> module=Phoenix.Endpoint.Supervisor function=log_access_url/2 line=435  [info] Access TpWeb.Endpoint at http://localhost:4000
Interactive Elixir (1.19.1) - press Ctrl+C to exit (type h() ENTER for help)

  ../priv/static/assets/app.js  5.0mb ⚠️

⚡ Done in 133ms

Rebuilding...

Done in 785ms.
iex(tp@127.0.0.1)1>

jozuas · October 29, 2025, 4:47pm

Yep, that was it. I removed kino from my deps, and with Elixir 1.19 OTP 28 I no longer get the stacktrace.

nicanor · October 29, 2025, 5:50pm

Same here.
I removed Kino from mix.exs and I’m not experiencing the error anymore.
Thanks for your help!

sorentwo · October 29, 2025, 6:09pm

Amazing to discover that it was a “mix recompile” hook within kino all along.

josevalim · October 29, 2025, 10:35pm

Great find!

Unfortunately I cannot reproduce the issue locally as well… so if anyone can isolate it, now that you know kino is the root cause, it would be very appreciated, because I don’t think it is caused by mix_recompile.

stevegrossi · October 30, 2025, 1:44pm

Interesting. The issue I’m running into must be different, because we’re not using kino either directly or transitively (it’s not in mix.lock). Notably, changing a heex file triggers live reloading but does not cause Oban to crash, while changing a js or css file also triggers live reloading and always causes Oban to crash (taking down the entire application) locally with (UndefinedFunctionError) function MyApp.Repo.query/3 is undefined (module MyApp.Repo is not available).

These are the logs with sasl reports enabled (I had to cut out some Oban processes shutting down in the middle to fit within the forum’s character limit):

Crash logs with sasl reports

[watch] build started (change: "js/my_app/app.js")
[watch] build finished
file=lib/phoenix_live_reload/channel.ex line=48 [debug] Live reload: priv/static/assets/my_app.js
Done in 326µs

[0.36ms] [@tailwindcss/cli] (watcher)
[0.31ms]   ↳ Scan for candidates

Compiling 47 files (.ex)
file=gen_server.erl line=2785 [error] GenServer {Oban.Registry, {Oban, Oban.Met.Listener}} terminating
** (UndefinedFunctionError) function MyApp.Repo.query/3 is undefined (module MyApp.Repo is not available)
    (my_app 0.0.1) MyApp.Repo.query("SELECT pg_notify($1, payload) FROM json_array_elements_text($2::json) AS payload", ["public.oban_metrics", ["H4sIAAAAAAAAE6tWykvMTVWyUvJPSsxT0lHKy08B8YJLUstSi3V9E5Od8vOzdQOK8oFyJZkglYbmZkBkYWpppqOUm1pSlJlcrGQVHVsLAHXHg3ZKAAAA"]], [prefix: "public", log: false, oban: true, telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Pro.Engines.Smart, get_dynamic_repo: nil, insert_trigger: true, log: false, name: Oban, node: "Steves-MacBook-Pro", notifier: {Oban.Notifiers.Postgres, []}, peer: {Oban.Peers.Database, []}, plugins: [{Oban.Pro.Plugins.DynamicPruner, [mode: {:max_age, {2, :days}}, limit: 3500, before_delete: {MyApp.Oban.JobsArchiver, :call, []}]}, {Oban.Pro.Plugins.DynamicLifeline, []}, {Oban.Plugins.Cron, [crontab: [{"0 * * * *", MyApp.BatchNotificationWorker, [args: %{}]}, {"0 2 * * *", MyApp.BatchNotificationCleanupWorker, [args: %{}]}, {"*/30 * * * *", MyApp.BatchWorker, [args: %{action: "delete_expired_batch_operations", scheduled: true}, unique: [period: 60, states: [:available, :scheduled, :executing]]]}, {"10 6 * * *", MyApp.CampaignsMetrics.CalculateMessageMetricsWorker}, {"0 6 * * *", MyApp.CampaignsMetrics.CalculateObjectiveCompletionsWorker}, {"@hourly", MyApp.CreateAbandonedActionsWorker, [args: %{}]}, {"15 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "import_new_members"}]}, {"0 8 * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_all_contacts"}]}, {"0 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_contacts_with_active_journeys"}]}, {"@hourly", MyApp.FactsRefreshWorker, [args: %{"job_type" => "account_opened_action_facts"}]}, {"@daily", MyApp.FactsRefreshWorker, ...}, ...]]}], ...}]])
    (oban 2.20.1) lib/oban/notifiers/postgres.ex:101: Oban.Notifiers.Postgres.notify/3
    (oban 2.20.1) lib/oban/notifier.ex:204: anonymous fn/4 in Oban.Notifier.notify/3
    (telemetry 1.3.0) /Users/steve/Work/monorepo/my_app/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
    (oban_met 1.0.3) lib/oban/met/listener.ex:82: Oban.Met.Listener.handle_info/2
    (stdlib 7.1) gen_server.erl:2434: :gen_server.try_handle_info/3
    (stdlib 7.1) gen_server.erl:2420: :gen_server.handle_msg/3
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Last message: :report
file=proc_lib.erl line=954 [error] Process #PID<0.1197.0> terminating
** (UndefinedFunctionError) function MyApp.Repo.query/3 is undefined (module MyApp.Repo is not available)
    (my_app 0.0.1) MyApp.Repo.query("SELECT pg_notify($1, payload) FROM json_array_elements_text($2::json) AS payload", ["public.oban_metrics", ["H4sIAAAAAAAAE6tWykvMTVWyUvJPSsxT0lHKy08B8YJLUstSi3V9E5Od8vOzdQOK8oFyJZkglYbmZkBkYWpppqOUm1pSlJlcrGQVHVsLAHXHg3ZKAAAA"]], [prefix: "public", log: false, oban: true, telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Pro.Engines.Smart, get_dynamic_repo: nil, insert_trigger: true, log: false, name: Oban, node: "Steves-MacBook-Pro", notifier: {Oban.Notifiers.Postgres, []}, peer: {Oban.Peers.Database, []}, plugins: [{Oban.Pro.Plugins.DynamicPruner, [mode: {:max_age, {2, :days}}, limit: 3500, before_delete: {MyApp.Oban.JobsArchiver, :call, []}]}, {Oban.Pro.Plugins.DynamicLifeline, []}, {Oban.Plugins.Cron, [crontab: [{"0 * * * *", MyApp.BatchNotificationWorker, [args: %{}]}, {"0 2 * * *", MyApp.BatchNotificationCleanupWorker, [args: %{}]}, {"*/30 * * * *", MyApp.BatchWorker, [args: %{action: "delete_expired_batch_operations", scheduled: true}, unique: [period: 60, states: [:available, :scheduled, :executing]]]}, {"10 6 * * *", MyApp.CampaignsMetrics.CalculateMessageMetricsWorker}, {"0 6 * * *", MyApp.CampaignsMetrics.CalculateObjectiveCompletionsWorker}, {"@hourly", MyApp.CreateAbandonedActionsWorker, [args: %{}]}, {"15 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "import_new_members"}]}, {"0 8 * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_all_contacts"}]}, {"0 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_contacts_with_active_journeys"}]}, {"@hourly", MyApp.FactsRefreshWorker, [args: %{"job_type" => "account_opened_action_facts"}]}, {"@daily", MyApp.FactsRefreshWorker, ...}, ...]]}], ...}]])
    (oban 2.20.1) lib/oban/notifiers/postgres.ex:101: Oban.Notifiers.Postgres.notify/3
    (oban 2.20.1) lib/oban/notifier.ex:204: anonymous fn/4 in Oban.Notifier.notify/3
    (telemetry 1.3.0) /Users/steve/Work/monorepo/my_app/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
    (oban_met 1.0.3) lib/oban/met/listener.ex:82: Oban.Met.Listener.handle_info/2
    (stdlib 7.1) gen_server.erl:2434: :gen_server.try_handle_info/3
    (stdlib 7.1) gen_server.erl:2420: :gen_server.handle_msg/3
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Initial Call: Oban.Met.Listener.init/1
Ancestors: [#PID<0.1171.0>, Oban.Met.Supervisor, #PID<0.439.0>]
Message Queue Length: 0
Messages: []
Links: [#PID<0.1171.0>, #PID<0.428.0>]
Dictionary: []
Trapping Exits: true
Status: :running
file=supervisor.erl line=1359 [error] Child {:via, Registry, {Oban.Registry, {Oban, Oban.Met.Listener}}} of Supervisor {Oban.Registry, {Oban, Oban.Met}} terminated
** (exit) an exception was raised:
    ** (UndefinedFunctionError) function MyApp.Repo.query/3 is undefined (module MyApp.Repo is not available)
        (my_app 0.0.1) MyApp.Repo.query("SELECT pg_notify($1, payload) FROM json_array_elements_text($2::json) AS payload", ["public.oban_metrics", ["H4sIAAAAAAAAE6tWykvMTVWyUvJPSsxT0lHKy08B8YJLUstSi3V9E5Od8vOzdQOK8oFyJZkglYbmZkBkYWpppqOUm1pSlJlcrGQVHVsLAHXHg3ZKAAAA"]], [prefix: "public", log: false, oban: true, telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Pro.Engines.Smart, get_dynamic_repo: nil, insert_trigger: true, log: false, name: Oban, node: "Steves-MacBook-Pro", notifier: {Oban.Notifiers.Postgres, []}, peer: {Oban.Peers.Database, []}, plugins: [{Oban.Pro.Plugins.DynamicPruner, [mode: {:max_age, {2, :days}}, limit: 3500, before_delete: {MyApp.Oban.JobsArchiver, :call, []}]}, {Oban.Pro.Plugins.DynamicLifeline, []}, {Oban.Plugins.Cron, [crontab: [{"0 * * * *", MyApp.BatchNotificationWorker, [args: %{}]}, {"0 2 * * *", MyApp.BatchNotificationCleanupWorker, [args: %{}]}, {"*/30 * * * *", MyApp.BatchWorker, [args: %{action: "delete_expired_batch_operations", scheduled: true}, unique: [period: 60, states: [:available, :scheduled, :executing]]]}, {"10 6 * * *", MyApp.CampaignsMetrics.CalculateMessageMetricsWorker}, {"0 6 * * *", MyApp.CampaignsMetrics.CalculateObjectiveCompletionsWorker}, {"@hourly", MyApp.CreateAbandonedActionsWorker, [args: %{}]}, {"15 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "import_new_members"}]}, {"0 8 * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_all_contacts"}]}, {"0 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_contacts_with_active_journeys"}]}, {"@hourly", MyApp.FactsRefreshWorker, [args: %{"job_type" => "account_opened_action_facts"}]}, {"@daily", MyApp.FactsRefreshWorker, ...}, ...]]}], ...}]])
        (oban 2.20.1) lib/oban/notifiers/postgres.ex:101: Oban.Notifiers.Postgres.notify/3
        (oban 2.20.1) lib/oban/notifier.ex:204: anonymous fn/4 in Oban.Notifier.notify/3
        (telemetry 1.3.0) /Users/steve/Work/monorepo/my_app/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
        (oban_met 1.0.3) lib/oban/met/listener.ex:82: Oban.Met.Listener.handle_info/2
        (stdlib 7.1) gen_server.erl:2434: :gen_server.try_handle_info/3
        (stdlib 7.1) gen_server.erl:2420: :gen_server.handle_msg/3
        (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Pid: #PID<0.1197.0>
Start Call: Oban.Met.Listener.start_link([name: {:via, Registry, {Oban.Registry, {Oban, Oban.Met.Listener}}}, conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Pro.Engines.Smart, get_dynamic_repo: nil, insert_trigger: true, log: false, name: Oban, node: "Steves-MacBook-Pro", notifier: {Oban.Notifiers.Postgres, []}, peer: {Oban.Peers.Database, []}, plugins: [{Oban.Pro.Plugins.DynamicPruner, [mode: {:max_age, {2, :days}}, limit: 3500, before_delete: {MyApp.Oban.JobsArchiver, :call, []}]}, {Oban.Pro.Plugins.DynamicLifeline, []}, {Oban.Plugins.Cron, [crontab: [{"0 * * * *", MyApp.BatchNotificationWorker, [args: %{}]}, {"0 2 * * *", MyApp.BatchNotificationCleanupWorker, [args: %{}]}, {"*/30 * * * *", MyApp.BatchWorker, [args: %{action: "delete_expired_batch_operations", scheduled: true}, unique: [period: 60, states: [:available, :scheduled, :executing]]]}, {"10 6 * * *", MyApp.CampaignsMetrics.CalculateMessageMetricsWorker}, {"0 6 * * *", MyApp.CampaignsMetrics.CalculateObjectiveCompletionsWorker}, {"@hourly", MyApp.CreateAbandonedActionsWorker, [args: %{}]}, {"15 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "import_new_members"}]}, {"0 8 * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_all_contacts"}]}, {"0 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_contacts_with_active_journeys"}]}, {"@hourly", MyApp.FactsRefreshWorker, [args: %{...}]}, ...]]}], ...}])
Restart: :permanent
Shutdown: 5000
Type: :worker
file=supervisor.erl line=2240 [info] Child {:via, Registry, {Oban.Registry, {Oban, Oban.Met.Listener}}} of Supervisor {Oban.Registry, {Oban, Oban.Met}} started
Pid: #PID<0.3883.0>
Start Call: Oban.Met.Listener.start_link([name: {:via, Registry, {Oban.Registry, {Oban, Oban.Met.Listener}}}, conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Pro.Engines.Smart, get_dynamic_repo: nil, insert_trigger: true, log: false, name: Oban, node: "Steves-MacBook-Pro", notifier: {Oban.Notifiers.Postgres, []}, peer: {Oban.Peers.Database, []}, plugins: [{Oban.Pro.Plugins.DynamicPruner, [mode: {:max_age, {2, :days}}, limit: 3500, before_delete: {MyApp.Oban.JobsArchiver, :call, []}]}, {Oban.Pro.Plugins.DynamicLifeline, []}, {Oban.Plugins.Cron, [crontab: [{"0 * * * *", MyApp.BatchNotificationWorker, [args: %{}]}, {"0 2 * * *", MyApp.BatchNotificationCleanupWorker, [args: %{}]}, {"*/30 * * * *", MyApp.BatchWorker, [args: %{action: "delete_expired_batch_operations", scheduled: true}, unique: [period: 60, states: [:available, :scheduled, :executing]]]}, {"10 6 * * *", MyApp.CampaignsMetrics.CalculateMessageMetricsWorker}, {"0 6 * * *", MyApp.CampaignsMetrics.CalculateObjectiveCompletionsWorker}, {"@hourly", MyApp.CreateAbandonedActionsWorker, [args: %{}]}, {"15 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "import_new_members"}]}, {"0 8 * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_all_contacts"}]}, {"0 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_contacts_with_active_journeys"}]}, {"@hourly", MyApp.FactsRefreshWorker, [args: %{...}]}, ...]]}], ...}])
Restart: :permanent
Shutdown: 5000
Type: :worker
file=gen_server.erl line=2785 [error] GenServer {Oban.Registry, {Oban, Oban.Stager}} terminating
** (UndefinedFunctionError) function MyApp.Repo.transaction/2 is undefined (module MyApp.Repo is not available)
    (my_app 0.0.1) MyApp.Repo.transaction(#Function<4.4575437/0 in Oban.Stager.stage_and_notify/2>, [prefix: "public", log: false, oban: true, telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Pro.Engines.Smart, get_dynamic_repo: nil, insert_trigger: true, log: false, name: Oban, node: "Steves-MacBook-Pro", notifier: {Oban.Notifiers.Postgres, []}, peer: {Oban.Peers.Database, []}, plugins: [{Oban.Pro.Plugins.DynamicPruner, [mode: {:max_age, {2, :days}}, limit: 3500, before_delete: {MyApp.Oban.JobsArchiver, :call, []}]}, {Oban.Pro.Plugins.DynamicLifeline, []}, {Oban.Plugins.Cron, [crontab: [{"0 * * * *", MyApp.BatchNotificationWorker, [args: %{}]}, {"0 2 * * *", MyApp.BatchNotificationCleanupWorker, [args: %{}]}, {"*/30 * * * *", MyApp.BatchWorker, [args: %{action: "delete_expired_batch_operations", scheduled: true}, unique: [period: 60, states: [:available, :scheduled, :executing]]]}, {"10 6 * * *", MyApp.CampaignsMetrics.CalculateMessageMetricsWorker}, {"0 6 * * *", MyApp.CampaignsMetrics.CalculateObjectiveCompletionsWorker}, {"@hourly", MyApp.CreateAbandonedActionsWorker, [args: %{}]}, {"15 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "import_new_members"}]}, {"0 8 * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_all_contacts"}]}, {"0 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_contacts_with_active_journeys"}]}, {"@hourly", MyApp.FactsRefreshWorker, [args: %{"job_type" => "account_opened_action_facts"}]}, {"@daily", MyApp.FactsRefreshWorker, ...}, ...]]}], ...}]])
    (oban 2.20.1) lib/oban/repo.ex:156: Oban.Repo.transaction/4
    (oban 2.20.1) lib/oban/stager.ex:82: Oban.Stager.stage_and_notify/2
    (oban 2.20.1) lib/oban/stager.ex:59: anonymous fn/2 in Oban.Stager.handle_info/2
    (telemetry 1.3.0) /Users/steve/Work/monorepo/my_app/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
    (oban 2.20.1) lib/oban/stager.ex:58: Oban.Stager.handle_info/2
    (stdlib 7.1) gen_server.erl:2434: :gen_server.try_handle_info/3
    (stdlib 7.1) gen_server.erl:2420: :gen_server.handle_msg/3
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Last message: :stage
file=gen_server.erl line=2785 [error] GenServer {Oban.Registry, {Oban, Oban.Met.Reporter}} terminating
** (UndefinedFunctionError) function MyApp.Repo.transaction/2 is undefined (module MyApp.Repo is not available)
    (my_app 0.0.1) MyApp.Repo.transaction(#Function<2.70649110/0 in Oban.Met.Reporter.checks/1>, [prefix: "public", log: false, oban: true, telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Pro.Engines.Smart, get_dynamic_repo: nil, insert_trigger: true, log: false, name: Oban, node: "Steves-MacBook-Pro", notifier: {Oban.Notifiers.Postgres, []}, peer: {Oban.Peers.Database, []}, plugins: [{Oban.Pro.Plugins.DynamicPruner, [mode: {:max_age, {2, :days}}, limit: 3500, before_delete: {MyApp.Oban.JobsArchiver, :call, []}]}, {Oban.Pro.Plugins.DynamicLifeline, []}, {Oban.Plugins.Cron, [crontab: [{"0 * * * *", MyApp.BatchNotificationWorker, [args: %{}]}, {"0 2 * * *", MyApp.BatchNotificationCleanupWorker, [args: %{}]}, {"*/30 * * * *", MyApp.BatchWorker, [args: %{action: "delete_expired_batch_operations", scheduled: true}, unique: [period: 60, states: [:available, :scheduled, :executing]]]}, {"10 6 * * *", MyApp.CampaignsMetrics.CalculateMessageMetricsWorker}, {"0 6 * * *", MyApp.CampaignsMetrics.CalculateObjectiveCompletionsWorker}, {"@hourly", MyApp.CreateAbandonedActionsWorker, [args: %{}]}, {"15 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "import_new_members"}]}, {"0 8 * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_all_contacts"}]}, {"0 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_contacts_with_active_journeys"}]}, {"@hourly", MyApp.FactsRefreshWorker, [args: %{"job_type" => "account_opened_action_facts"}]}, {"@daily", MyApp.FactsRefreshWorker, ...}, ...]]}], ...}]])
    (oban 2.20.1) lib/oban/repo.ex:156: Oban.Repo.transaction/4
    (oban_met 1.0.3) lib/oban/met/reporter.ex:98: Oban.Met.Reporter.handle_info/2
    (stdlib 7.1) gen_server.erl:2434: :gen_server.try_handle_info/3
    (stdlib 7.1) gen_server.erl:2420: :gen_server.handle_msg/3
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Last message: :checkpoint
file=proc_lib.erl line=954 [error] Process #PID<0.1163.0> terminating
** (UndefinedFunctionError) function MyApp.Repo.transaction/2 is undefined (module MyApp.Repo is not available)
    (my_app 0.0.1) MyApp.Repo.transaction(#Function<4.4575437/0 in Oban.Stager.stage_and_notify/2>, [prefix: "public", log: false, oban: true, telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Pro.Engines.Smart, get_dynamic_repo: nil, insert_trigger: true, log: false, name: Oban, node: "Steves-MacBook-Pro", notifier: {Oban.Notifiers.Postgres, []}, peer: {Oban.Peers.Database, []}, plugins: [{Oban.Pro.Plugins.DynamicPruner, [mode: {:max_age, {2, :days}}, limit: 3500, before_delete: {MyApp.Oban.JobsArchiver, :call, []}]}, {Oban.Pro.Plugins.DynamicLifeline, []}, {Oban.Plugins.Cron, [crontab: [{"0 * * * *", MyApp.BatchNotificationWorker, [args: %{}]}, {"0 2 * * *", MyApp.BatchNotificationCleanupWorker, [args: %{}]}, {"*/30 * * * *", MyApp.BatchWorker, [args: %{action: "delete_expired_batch_operations", scheduled: true}, unique: [period: 60, states: [:available, :scheduled, :executing]]]}, {"10 6 * * *", MyApp.CampaignsMetrics.CalculateMessageMetricsWorker}, {"0 6 * * *", MyApp.CampaignsMetrics.CalculateObjectiveCompletionsWorker}, {"@hourly", MyApp.CreateAbandonedActionsWorker, [args: %{}]}, {"15 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "import_new_members"}]}, {"0 8 * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_all_contacts"}]}, {"0 * * * *", MyApp.CmcFlex.DelegationWorker, [args: %{"delegate_job_type" => "update_contacts_with_active_journeys"}]}, {"@hourly", MyApp.FactsRefreshWorker, [args: %{"job_type" => "account_opened_action_facts"}]}, {"@daily", MyApp.FactsRefreshWorker, ...}, ...]]}], ...}]])
    (oban 2.20.1) lib/oban/repo.ex:156: Oban.Repo.transaction/4
    (oban 2.20.1) lib/oban/stager.ex:82: Oban.Stager.stage_and_notify/2
    (oban 2.20.1) lib/oban/stager.ex:59: anonymous fn/2 in Oban.Stager.handle_info/2
    (telemetry 1.3.0) /Users/steve/Work/monorepo/my_app/deps/telemetry/src/telemetry.erl:324: :telemetry.span/3
    (oban 2.20.1) lib/oban/stager.ex:58: Oban.Stager.handle_info/2
    (stdlib 7.1) gen_server.erl:2434: :gen_server.try_handle_info/3
    (stdlib 7.1) gen_server.erl:2420: :gen_server.handle_msg/3
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Initial Call: Oban.Stager.init/1
Ancestors: [#PID<0.1162.0>, #PID<0.1029.0>, MyApp.Supervisor, #PID<0.601.0>]
Message Queue Length: 0
Messages: []
Links: [#PID<0.1162.0>, #PID<0.428.0>]
Dictionary: []
Trapping Exits: true
Status: :running
file=proc_lib.erl line=954 [error] Process #PID<0.1202.0> terminating

... many Oban processes crashing ...

file=gen_server.erl line=2785 [error] GenServer #PID<0.991.0> terminating
** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
    (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, MyAppWeb.AdminSocket})
    (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
    (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
    (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.720.0>, :shutdown}
file=proc_lib.erl line=954 [error] Process #PID<0.991.0> terminating
** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
    (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, MyAppWeb.AdminSocket})
    (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
    (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
    (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Initial Call: Phoenix.Socket.PoolDrainer.init/1
Ancestors: [MyAppWeb.Endpoint, MyApp.Supervisor, #PID<0.601.0>]
Message Queue Length: 0
Messages: []
Links: [#PID<0.720.0>]
Dictionary: []
Trapping Exits: true
Status: :running
file=supervisor.erl line=1529 [error] Child {:terminator, MyAppWeb.AdminSocket} of Supervisor MyAppWeb.Endpoint shut down abnormally
** (exit) an exception was raised:
    ** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
        (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, MyAppWeb.AdminSocket})
        (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
        (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
        (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
        (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Pid: #PID<0.991.0>
Start Call: Phoenix.Socket.PoolDrainer.start_link({MyAppWeb.Endpoint, MyAppWeb.AdminSocket, [endpoint: MyAppWeb.Endpoint, websocket: [check_origin: {MyApp.Origin, :check_origin?, []}], longpoll: false, drainer: []]})
Restart: :permanent
Shutdown: 30000
Type: :worker
file=gen_server.erl line=2785 [error] GenServer #PID<0.990.0> terminating
** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
    (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, MyAppWeb.PublicSocket})
    (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
    (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
    (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.720.0>, :shutdown}
file=proc_lib.erl line=954 [error] Process #PID<0.990.0> terminating
** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
    (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, MyAppWeb.PublicSocket})
    (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
    (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
    (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Initial Call: Phoenix.Socket.PoolDrainer.init/1
Ancestors: [MyAppWeb.Endpoint, MyApp.Supervisor, #PID<0.601.0>]
Message Queue Length: 0
Messages: []
Links: [#PID<0.720.0>]
Dictionary: []
Trapping Exits: true
Status: :running
file=supervisor.erl line=1529 [error] Child {:terminator, MyAppWeb.PublicSocket} of Supervisor MyAppWeb.Endpoint shut down abnormally
** (exit) an exception was raised:
    ** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
        (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, MyAppWeb.PublicSocket})
        (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
        (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
        (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
        (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Pid: #PID<0.990.0>
Start Call: Phoenix.Socket.PoolDrainer.start_link({MyAppWeb.Endpoint, MyAppWeb.PublicSocket, [endpoint: MyAppWeb.Endpoint, websocket: [check_origin: {MyApp.Origin, :check_origin?, []}], longpoll: false, drainer: []]})
Restart: :permanent
Shutdown: 30000
Type: :worker
file=gen_server.erl line=2785 [error] GenServer #PID<0.989.0> terminating
** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
    (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, Phoenix.LiveView.Socket})
    (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
    (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
    (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.720.0>, :shutdown}
file=proc_lib.erl line=954 [error] Process #PID<0.989.0> terminating
** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
    (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, Phoenix.LiveView.Socket})
    (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
    (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
    (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Initial Call: Phoenix.Socket.PoolDrainer.init/1
Ancestors: [MyAppWeb.Endpoint, MyApp.Supervisor, #PID<0.601.0>]
Message Queue Length: 0
Messages: []
Links: [#PID<0.720.0>]
Dictionary: []
Trapping Exits: true
Status: :running
file=supervisor.erl line=1529 [error] Child {:terminator, Phoenix.LiveView.Socket} of Supervisor MyAppWeb.Endpoint shut down abnormally
** (exit) an exception was raised:
    ** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
        (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, Phoenix.LiveView.Socket})
        (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
        (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
        (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
        (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Pid: #PID<0.989.0>
Start Call: Phoenix.Socket.PoolDrainer.start_link({MyAppWeb.Endpoint, Phoenix.LiveView.Socket, [endpoint: MyAppWeb.Endpoint, websocket: [connect_info: [:uri, {:session, [store: :cookie, key: "_my_app_key", signing_salt: "yot42ivA"]}]], drainer: []]})
Restart: :permanent
Shutdown: 30000
Type: :worker
file=gen_server.erl line=2785 [error] GenServer #PID<0.988.0> terminating
** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
    (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, Phoenix.LiveReloader.Socket})
    (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
    (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
    (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.720.0>, :shutdown}
file=proc_lib.erl line=954 [error] Process #PID<0.988.0> terminating
** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
    (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, Phoenix.LiveReloader.Socket})
    (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
    (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
    (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
    (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Initial Call: Phoenix.Socket.PoolDrainer.init/1
Ancestors: [MyAppWeb.Endpoint, MyApp.Supervisor, #PID<0.601.0>]
Message Queue Length: 0
Messages: []
Links: [#PID<0.720.0>]
Dictionary: []
Trapping Exits: true
Status: :running
file=supervisor.erl line=1529 [error] Child {:terminator, Phoenix.LiveReloader.Socket} of Supervisor MyAppWeb.Endpoint shut down abnormally
** (exit) an exception was raised:
    ** (UndefinedFunctionError) function MyAppWeb.Endpoint.config/1 is undefined (module MyAppWeb.Endpoint is not available)
        (my_app 0.0.1) MyAppWeb.Endpoint.config({:socket, Phoenix.LiveReloader.Socket})
        (phoenix 1.8.1) lib/phoenix/socket/pool_supervisor.ex:97: Phoenix.Socket.PoolDrainer.terminate/2
        (stdlib 7.1) gen_server.erl:2482: :gen_server.try_terminate/3
        (stdlib 7.1) gen_server.erl:2733: :gen_server.terminate/9
        (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Pid: #PID<0.988.0>
Start Call: Phoenix.Socket.PoolDrainer.start_link({MyAppWeb.Endpoint, Phoenix.LiveReloader.Socket, [log: false, endpoint: MyAppWeb.Endpoint, drainer: []]})
Restart: :permanent
Shutdown: 30000
Type: :worker
Generated my_app app
file=lib/my_app_web/plugs/logger.ex line=32 [info] GET /login
file=lib/phoenix/logger.ex line=306 [debug] Processing with MyAppWeb.Jimi.LoginLive.__live__/0
  Parameters: %{}
  Pipelines: [:browser, :my_app]
file=lib/my_app_web/plugs/logger.ex line=45 [info] Sent 500 in 11ms
file=proc_lib.erl line=954 [error] Process #PID<0.3680.0> terminating
** (exit) {{{%RuntimeError{message: "could not lookup Ecto repo MyApp.Repo because it was not started or it does not exist"}, [{Ecto.Repo.Registry, :lookup, 1, [file: ~c"lib/ecto/repo/registry.ex", line: 22, error_info: %{module: Exception}]}, {Ecto.Repo.Supervisor, :tuplet, 2, [file: ~c"lib/ecto/repo/supervisor.ex", line: 176]}, {MyApp.Repo, :get_by, 3, [file: ~c"lib/my_app/repo.ex", line: 4]}, {MyApp.Teams, :load_team_from_subdomain, 1, [file: ~c"lib/my_app/teams/teams.ex", line: 697]}, {MyAppWeb.Plugs.LoadTeamFromSubdomain, :call, 2, [file: ~c"lib/my_app_web/plugs/load_team_from_subdomain.ex", line: 16]}, {MyAppWeb.Router, :my_app, 2, []}, {MyAppWeb.Router, :__pipe_through1__, 1, [file: ~c"lib/my_app_web/router.ex", line: 1]}, {Phoenix.Router, :__call__, 5, [file: ~c"lib/phoenix/router.ex", line: 407]}, {MyAppWeb.Endpoint, :plug_builder_call, 2, [file: ~c"lib/my_app_web/endpoint.ex", line: 1]}, {MyAppWeb.Endpoint, :"call (overridable 3)", 2, [file: ~c"lib/my_app_web/endpoint.ex", line: 1]}, {MyAppWeb.Endpoint, :"call (overridable 4)", 2, [file: ~c"deps/plug/lib/plug/debugger.ex", line: 155]}, {MyAppWeb.Endpoint, :call, 2, [file: ~c"lib/my_app_web/endpoint.ex", line: 1]}, {Phoenix.Endpoint.SyncCodeReloadPlug, :do_call, 4, [file: ~c"lib/phoenix/endpoint/sync_code_reload_plug.ex", line: 22]}, {Plug.Cowboy.Handler, :init, 2, ...}, ...]}, ...}, ...}
Initial Call: :cowboy_stream_h.request_process/3
Ancestors: [#PID<0.3679.0>, #PID<0.804.0>, #PID<0.785.0>, #PID<0.784.0>, #PID<0.782.0>, MyAppWeb.Endpoint, MyApp.Supervisor, #PID<0.601.0>]
Message Queue Length: 1
Messages: [plug_conn: :sent]
Links: [#PID<0.3679.0>]
Dictionary: [plug_csrf_token_per_host: %{secret_key_base: "NnfGOrfQWhOmp/NkIywhKIF3VSYFSMvzr+ykzanrrAC3MGzZTgE2eMF6gzPMiUAw"}, "$logger_metadata$": %{__sentry__: %{request: %{data: %{}, env: %{"REMOTE_ADDR" => "127.0.0.1", "REMOTE_PORT" => nil, "REQUEST_ID" => nil, "SERVER_NAME" => "dev.localhost", "SERVER_PORT" => 4000}, url: "http://dev.localhost:4000/login", headers: %{"accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-encoding" => "gzip, deflate, br, zstd", "accept-language" => "en-US,en;q=0.9", "cache-control" => "max-age=0", "connection" => "keep-alive", "host" => "dev.localhost:4000", "referer" => "http://dev.localhost:4000/login", "sec-ch-ua" => "\"Google Chrome\";v=\"141\", \"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"141\"", "sec-ch-ua-mobile" => "?0", "sec-ch-ua-platform" => "\"macOS\"", "sec-fetch-dest" => "document", "sec-fetch-mode" => "navigate", "sec-fetch-site" => "same-origin", "upgrade-insecure-requests" => "1", "user-agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"}, cookies: %{}, method: "GET", query_string: ""}, tags: %{team: "dev", route: "login", team_source: "referer subdomain"}}}]
Trapping Exits: false
Status: :running
[error] #PID<0.3680.0> running Phoenix.Endpoint.SyncCodeReloadPlug (connection #PID<0.3679.0>, stream id 1) terminated
Server: dev.localhost:4000 (http)
Request: GET /login
** (exit) an exception was raised:
    ** (RuntimeError) could not lookup Ecto repo MyApp.Repo because it was not started or it does not exist
        (ecto 3.13.4) lib/ecto/repo/registry.ex:22: Ecto.Repo.Registry.lookup/1
        (ecto 3.13.4) lib/ecto/repo/supervisor.ex:176: Ecto.Repo.Supervisor.tuplet/2
        (my_app 0.0.1) lib/my_app/repo.ex:4: MyApp.Repo.get_by/3
        (my_app 0.0.1) lib/my_app/teams/teams.ex:697: MyApp.Teams.load_team_from_subdomain/1
        (my_app 0.0.1) lib/my_app_web/plugs/load_team_from_subdomain.ex:16: MyAppWeb.Plugs.LoadTeamFromSubdomain.call/2
        (my_app 0.0.1) MyAppWeb.Router.my_app/2
        (my_app 0.0.1) lib/my_app_web/router.ex:1: MyAppWeb.Router.__pipe_through1__/1
        (phoenix 1.8.1) lib/phoenix/router.ex:407: Phoenix.Router.__call__/5
        (my_app 0.0.1) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.plug_builder_call/2
        (my_app 0.0.1) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint."call (overridable 3)"/2
        (my_app 0.0.1) deps/plug/lib/plug/debugger.ex:155: MyAppWeb.Endpoint."call (overridable 4)"/2
        (my_app 0.0.1) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.call/2
        (phoenix 1.8.1) lib/phoenix/endpoint/sync_code_reload_plug.ex:22: Phoenix.Endpoint.SyncCodeReloadPlug.do_call/4
        (plug_cowboy 2.7.4) lib/plug/cowboy/handler.ex:11: Plug.Cowboy.Handler.init/2
        (cowboy 2.14.2) /Users/steve/Work/monorepo/my_app/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2
        (cowboy 2.14.2) /Users/steve/Work/monorepo/my_app/deps/cowboy/src/cowboy_stream_h.erl:310: :cowboy_stream_h.execute/3
        (cowboy 2.14.2) /Users/steve/Work/monorepo/my_app/deps/cowboy/src/cowboy_stream_h.erl:299: :cowboy_stream_h.request_process/3
        (stdlib 7.1) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
file=supervisor.erl line=1529 [error] Child Plug.Cowboy.Drainer of Supervisor MyAppWeb.Endpoint shut down abnormally
** (exit) killed
Pid: #PID<0.987.0>
Start Call: Plug.Cowboy.Drainer.start_link([refs: [MyAppWeb.Endpoint.HTTP]])
Restart: :permanent
Shutdown: 5000
Type: :worker
file=application_controller.erl line=2129 [notice] Application my_app exited: shutdown

stevegrossi · October 30, 2025, 2:03pm

Since MyApp.Repo seems to be the epicenter, and after reading the linked issue Larger xref module cycles after upgrading from 1.18 -> 1.19 · Issue #14844 · elixir-lang/elixir · GitHub, I checked mix xref graph --source lib/my_app/repo.ex and noticed something surprising:

On Elixir 1.18 with otp28, MyApp.Repo has a cycle of length 26
On Elixir 1.19 with otp28, MyApp.Repo triggers WARNING: Source lib/gondor/repo.ex is part of a cycle of 481 nodes and this cycle has a compile dependency. Therefore source and the whole cycle will recompile whenever any of the files they depend on change.

It sounds like the problem may be due to 481 files needing to recompile due to that much-longer long compilation dependency cycle and perhaps MyApp.Repo isn’t ready by the time Oban looks for it?

jozuas · November 12, 2025, 4:43pm

That’s what I thought as well, but José explained that larger cycles are expected:

github.com/elixir-lang/elixir

Larger xref module cycles after upgrading from 1.18 -> 1.19

opened 05:22PM - 18 Oct 25 UTC

closed 12:27PM - 21 Oct 25 UTC

vanderhoop

### Elixir and Erlang/OTP versions ``` Erlang/OTP 27 [erts-15.2.6] [source] [64…-bit] [smp:14:14] [ds:14:14:10] [async-threads:1] [jit] Elixir 1.19.0 (compiled with Erlang/OTP 27) ``` ### Operating system Both macOS 15.6.1 and ubuntu-latest (on GH actions) ### Current behavior After upgrading from Elixir 1.18.4 -> 1.19.0, I'm now getting lengthier `xref` module cycles despite no changes to the application source code. For example, where before `mix xref graph --format cycles --min-cycle-size 5 --fail-above 0` identified no cycles, elixir 1.19.0 flags multiple cycles, one as long as 21 modules: ``` $ mix xref graph --format cycles --min-cycle-size 5 --fail-above 0 Compiling 1 file (.ex) Generated rum_dash app 2 cycles found. Showing them in decreasing size: Cycle of length 21: lib/rum_dash_web/components/app_layout.ex lib/rum_dash_web/components/layouts.ex lib/rum_dash_web/controllers/blog_controller.ex lib/rum_dash_web/controllers/client_error_controller.ex lib/rum_dash_web/controllers/email_previews_controller.ex lib/rum_dash_web/controllers/error_html.ex lib/rum_dash_web/controllers/fallback_controller.ex lib/rum_dash_web/controllers/health_check_controller.ex lib/rum_dash_web/controllers/navigation_controller.ex lib/rum_dash_web/controllers/oauth_controller.ex lib/rum_dash_web/controllers/page_controller.ex (export) lib/rum_dash_web/controllers/user_session_controller.ex lib/rum_dash_web/emails/daily_digest_template.ex lib/rum_dash_web/endpoint.ex lib/rum_dash_web/live/website_live/common.ex lib/rum_dash_web/plugs/admin_layout_plug.ex lib/rum_dash_web/plugs/early_hints_plug.ex lib/rum_dash_web/plugs/layout_plug.ex lib/rum_dash_web/plugs/website_plugs.ex lib/rum_dash_web/router.ex (export) lib/rum_dash_web/user_auth.ex Cycle of length 5: lib/rum_dash/accounts/user.ex lib/rum_dash/accounts/website.ex lib/rum_dash/accounts/website_membership.ex lib/rum_dash/analytics/navigation.ex lib/rum_dash/analytics/path_regex.ex ** (Mix) Too many cycles (found: 2, permitted: 0) ``` ### Expected behavior My (perhaps hopeful) expectation is that the cycle length should be unchanged when upgrading. That said, I know that Elixir v1.19 introduces some additional goodies to the compiler, so if those goodies depend on longer cycles, feel free to close.