Questions & Help>Questions

Oban bringing app down with `Repo not available` error In development

/phoenix/oban#gettext
johantell

johantell

Oban bringing app down with `Repo not available` error In development

We’ve come across a somewhat frustrating problem on our development machines where the application gets shut down after Oban fails with the following error:

** (UndefinedFunctionError) function MyApp.Repo.transaction/2 is undefined (module MyApp.Repo is not available)
    (my_app 0.1.0) MyApp.Repo.transaction(#Function<0.22423664/0 in Oban.Stager.check_leadership_and_stage/1>, [log: false, prefix: "public", telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}]])

I’ve been successfully able to reproduce (in our) it by:

  1. Starting the phoenix app with Oban
  2. Running mix gettext.extract --merge
  3. Reloading the page

When reloading the page the application stops responding and the full console output is this:

Compiling 424 files (.ex)
[error] GenServer {Oban.Registry, {Oban, Oban.Stager}} terminating
** (UndefinedFunctionError) function MyApp.Repo.transaction/2 is undefined (module MyApp.Repo is not available)
    (my_app 0.1.0) MyApp.Repo.transaction(#Function<0.22423664/0 in Oban.Stager.check_leadership_and_stage/1>, [log: false, prefix: "public", telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}]])
    (oban 2.14.2) lib/oban/stager.ex:86: anonymous fn/2 in Oban.Stager.handle_info/2
    (telemetry 0.4.3) /Users/johantell/Projects/my_app_elixir/deps/telemetry/src/telemetry.erl:272: :telemetry.span/3
    (oban 2.14.2) lib/oban/stager.ex:85: Oban.Stager.handle_info/2
    (stdlib 4.2) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.2) gen_server.erl:1200: :gen_server.handle_msg/6
    (stdlib 4.2) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: :stage
State: %Oban.Stager.State{conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}, name: {:via, Registry, {Oban.Registry, {Oban, Oban.Stager}}}, timer: #Reference<0.3262115946.2793144325.226216>, interval: 1000, limit: 5000, mode: :global, ping_at_tick: 60, swap_at_tick: 65, tick: 40}
[error] GenServer {Oban.Registry, {Oban, Oban.Stager}} terminating
** (UndefinedFunctionError) function MyApp.Repo.query/3 is undefined (module MyApp.Repo is not available)
    (my_app 0.1.0) MyApp.Repo.query("SELECT pg_notify($1, payload) FROM json_array_elements_text($2::json) AS payload", ["public.oban_stager", ["{\"ping\":\"pong\"}"]], [log: false, prefix: "public", telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}]])
    (oban 2.14.2) lib/oban/notifiers/postgres.ex:100: Oban.Notifiers.Postgres.notify/3
    (oban 2.14.2) lib/oban/notifier.ex:188: anonymous fn/2 in Oban.Notifier.with_span/4
    (telemetry 0.4.3) /Users/johantell/Projects/my_app_elixir/deps/telemetry/src/telemetry.erl:272: :telemetry.span/3
    (oban 2.14.2) lib/oban/stager.ex:178: Oban.Stager.check_notify_mode/1
    (oban 2.14.2) lib/oban/stager.ex:69: Oban.Stager.handle_continue/2
    (stdlib 4.2) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.2) gen_server.erl:865: :gen_server.loop/7
    (stdlib 4.2) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {:continue, :start}
State: %Oban.Stager.State{conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}, name: {:via, Registry, {Oban.Registry, {Oban, Oban.Stager}}}, timer: nil, interval: 1000, limit: 5000, mode: :global, ping_at_tick: 0, swap_at_tick: 5, tick: 0}
[error] GenServer {Oban.Registry, {Oban, Oban.Stager}} terminating
** (UndefinedFunctionError) function MyApp.Repo.query/3 is undefined (module MyApp.Repo is not available)
    (my_app 0.1.0) MyApp.Repo.query("SELECT pg_notify($1, payload) FROM json_array_elements_text($2::json) AS payload", ["public.oban_stager", ["{\"ping\":\"pong\"}"]], [log: false, prefix: "public", telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}]])
    (oban 2.14.2) lib/oban/notifiers/postgres.ex:100: Oban.Notifiers.Postgres.notify/3
    (oban 2.14.2) lib/oban/notifier.ex:188: anonymous fn/2 in Oban.Notifier.with_span/4
    (telemetry 0.4.3) /Users/johantell/Projects/my_app_elixir/deps/telemetry/src/telemetry.erl:272: :telemetry.span/3
    (oban 2.14.2) lib/oban/stager.ex:178: Oban.Stager.check_notify_mode/1
    (oban 2.14.2) lib/oban/stager.ex:69: Oban.Stager.handle_continue/2
    (stdlib 4.2) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.2) gen_server.erl:865: :gen_server.loop/7
    (stdlib 4.2) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {:continue, :start}
State: %Oban.Stager.State{conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}, name: {:via, Registry, {Oban.Registry, {Oban, Oban.Stager}}}, timer: nil, interval: 1000, limit: 5000, mode: :global, ping_at_tick: 0, swap_at_tick: 5, tick: 0}
[error] GenServer {Oban.Registry, {Oban, Oban.Stager}} terminating
** (UndefinedFunctionError) function MyApp.Repo.query/3 is undefined (module MyApp.Repo is not available)
    (my_app 0.1.0) MyApp.Repo.query("SELECT pg_notify($1, payload) FROM json_array_elements_text($2::json) AS payload", ["public.oban_stager", ["{\"ping\":\"pong\"}"]], [log: false, prefix: "public", telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}]])
    (oban 2.14.2) lib/oban/notifiers/postgres.ex:100: Oban.Notifiers.Postgres.notify/3
    (oban 2.14.2) lib/oban/notifier.ex:188: anonymous fn/2 in Oban.Notifier.with_span/4
    (telemetry 0.4.3) /Users/johantell/Projects/my_app_elixir/deps/telemetry/src/telemetry.erl:272: :telemetry.span/3
    (oban 2.14.2) lib/oban/stager.ex:178: Oban.Stager.check_notify_mode/1
    (oban 2.14.2) lib/oban/stager.ex:69: Oban.Stager.handle_continue/2
    (stdlib 4.2) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.2) gen_server.erl:865: :gen_server.loop/7
    (stdlib 4.2) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {:continue, :start}
State: %Oban.Stager.State{conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}, name: {:via, Registry, {Oban.Registry, {Oban, Oban.Stager}}}, timer: nil, interval: 1000, limit: 5000, mode: :global, ping_at_tick: 0, swap_at_tick: 5, tick: 0}
[error] GenServer {Oban.Registry, {Oban, Oban.Peer}} terminating
** (UndefinedFunctionError) function MyApp.Repo.transaction/2 is undefined (module MyApp.Repo is not available)
    (my_app 0.1.0) MyApp.Repo.transaction(#Function<1.78375590/0 in Oban.Peers.Postgres.terminate/2>, [log: false, prefix: "public", telemetry_options: [oban_conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}]])
    (oban 2.14.2) lib/oban/peers/postgres.ex:68: Oban.Peers.Postgres.terminate/2
    (stdlib 4.2) gen_server.erl:1161: :gen_server.try_terminate/3
    (stdlib 4.2) gen_server.erl:1351: :gen_server.terminate/10
    (stdlib 4.2) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.17657.0>, :shutdown}
State: %Oban.Peers.Postgres.State{conf: %Oban.Config{dispatch_cooldown: 5, engine: Oban.Engines.Basic, get_dynamic_repo: nil, log: false, name: Oban, node: "t@localhost", notifier: Oban.Notifiers.Postgres, peer: Oban.Peers.Postgres, plugins: [{Oban.Plugins.Cron, [crontab: [{"0 9-17 * * *", MyApp.ScheduledJob, [queue: :custify]}]]}, {Oban.Plugins.Pruner, []}], prefix: "public", queues: [default: [limit: 10], mailers: [limit: 20], webhooks: [limit: 5], custify: [limit: 1]], repo: MyApp.Repo, shutdown_grace_period: 15000, stage_interval: 1000, testing: :disabled}, name: {:via, Registry, {Oban.Registry, {Oban, Oban.Peer}}}, timer: #Reference<0.3262115946.2793144323.228688>, interval: 30000, leader?: true, leader_boost: 2}

[notice] Application my_app exited: shutdown

I sadly have to little knowledge about how the code reloader and compiler handles when files are recompiled like this but we only started seeing the problems after upgrading to Oban 2.14.x.

Anyone has experienced this or have any good ideas on how we can try to sort it out?

35 comments
/phoenix/oban#gettext
2231 35

Marked As Solved

sorentwo

sorentwo

Oban Core Team

Recompilation in development mode causes a lot of issues with module references. Do you have dependencies that cause your Repo to recompile frequently? Does anything show up for it in xref?

All Repo calls are wrapped by Oban.Repo, which supports Ecto’s dynamic repos. Maybe you can try setting get_dynamic_repo and have it return a static repo?

config :my_app, Oban,
  repo: MyApp.Repo,
  get_dynamic_repo: fn -> MyApp.Repo end,
  ...
0
Post #17

Also Liked

vkryukov

vkryukov

Not sure if it’s related, but there was a thread recently with similar symptoms where kino dependency was found to be a culprit: Recompilation triggered after request in dev even without code changes (iex -S mix phx.server) - #9 by axelson

3
Post #28
jozuas

jozuas

Yep, that was it. I removed kino from my deps, and with Elixir 1.19 OTP 28 I no longer get the stacktrace.

2
Post #30
nicanor

nicanor

Same here.
I removed Kino from mix.exs and I’m not experiencing the error anymore.
Thanks for your help!

2
Post #31

Last Post!

jozuas

jozuas

That’s what I thought as well, but José explained that larger cycles are expected:
https://github.com/elixir-lang/elixir/issues/14844#issuecomment-3426332117

1
Post #36

Where Next?

View thread on forum (has 35 responses!)
phoenix
oban
gettext
Home Questions & Help>Questions
/phoenix/oban#gettext
16 2231 35
Last post

Popular in Questions Top

vegabook
Questions & Help>Questions

How to serve an img with Phoenix?
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
/phoenix
15 18833 5
New
Emily
Questions & Help>Questions

How To Get Phoenix & VueJS working Together?
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
/phoenix
93 22912 42
New
jononomo
Questions & Help>Questions

Websocket connection works on localhost, but get 403 error when deployed via docker
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
/phoenix#channels
8 27138 12
New
dokuzbir
Questions & Help>Questions

Visual Studio Code - how to highlight html closing tags in html.eex?
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
/phoenix#eex#troubleshooting#code-editors
12 21205 3
New
sen
Questions & Help>Questions

How to set environment variables in dev.exs?
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
/phoenix
31 22221 15
New
siddhant3030
Questions & Help>Questions

How can I write a raw sql query?
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
/phoenix#ecto
13 19876 20
New
svb
Questions & Help>Questions

Form submit on Enter keypress for textarea input type
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
/phoenix#liveview
1 13733 6
New

Other popular topics Top

Qqwy
Blogs & Podcasts>Blog Posts

Elixir Blog Posts
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
hexdocs.pm
#blog-posts#wiki#stickies#official-blog-posts-thread
3271 130286 1222
New
gshaw
Questions & Help>Questions

Idiomatic guard clause for checking not nil
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
#guard-clauses
13 43711 3
New
bsollish-terakeet
Questions & Help>Questions

Checking if an enum is empty - Credo vs Compiler
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
#compiler#credo
8 23323 6
New
WestKeys
Questions & Help>Questions

Mint vs Finch vs Gun vs Tesla vs HTTPoison etc
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
#http_client
253 22752 30
New
jason.o
Questions & Help>Questions

No such input `xxxxx` for action ResourceName1.create
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
/ash
3 78754 2
New
AstonJ
Chat & Discussions>Polls

Which Neovim configuration distro do you use/recommend? (Poll)
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
#polls#code-editors#neovim
79 14879 31
New
Back to top

Latest Phoenix Threads

Phoenix Forum ❯

Latest Oban Threads

Oban Forum ❯

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking for Elixir devs who love to ship. Start your free account.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking for Elixir devs who love to ship. Start your free account.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement