Oban: How to resume a paused queue on startup

Questions / Help
1

During application start up after Supervisor.start_link(children, opts) sets up the supervision tree with all attendant processes I want to resume an Oban queue I configured to be paused by default.

The use case is that in my deployment I have a migration phase and a service startup phase. The paused queue should only be resumed in the service startup phase. Each phase uses the same Docker image to boot (a precompiled Elixir release). The migration phase uses this container for a Kubernetes initContainer to do database migrations and other initialization task. The service startup phase uses the same container image to run an Elixir service.

Since the migration container terminates prior to some async jobs finishing, effectively giving false positives for job completion, we changed our approach to save the async jobs immediately to the DB with Oban to the paused queue. In the next phase the service deployment resumes that queue and process all the jobs so we can be more sure the jobs will complete.

An environment variable controls whether or not to resume this job processing. That variable is false in the migration environment and true in the service environment. When the variable is true I want the queue to be resumed, like the following:

defmodule MyApp.Application do
  @my_env_var Application.fetch_env!(:myapp, :some_var)
  def start(_, _) do
    topologies ...
    children ...
    Supervisor.start_link(children, opts)
    if @my_env_var do
      Oban.resume_queue(queue: :my queue)
    end
  end
end

and in releases.exs

import Config
config: :myapp, 
  myvar: System.get_env("PERFORM_JOB")

Is this how it is supposed to work?

2

You are using module attribute which is inlined by compilation. (is inline correct term tho i dont know) Basically the env when you are building release is used not the one from releases.ex

Use directly application.get_env

I wonder why do you even need to boot up whole app to run migrations? I was convinced that only Repo supervisor is started.
(written on phone)

3

The reason we boot the whole app is so we can use exactly the same image for migration that we do for running the app which means less artifacts to manage. We could provide different startup parameters.

Application.get_env is a non starter because it pulls variables from compile time rather than run time.

4

So does fetch env doesnt it? I meant fetch env, use it directly not through module attribute.

You don’t need to start your otp app to run migrations?

https://hexdocs.pm/phoenix/releases.html#ecto-migrations-and-custom-commands In this example it’s only loaded not started

5

Is the goal to only pause/resume some queues rather than all the queues? I don’t see why you’d want any of them running in a migration instance. I’d suggest configuring queues: false to avoid running queues entirely.

This is a variant of what is described in the Splitting Queues Guide. The difference is that you wouldn’t start any queues if the environment variable was “false”.

2 Likes
6

Yes, you’re right. After some further thought we agreed with you, we don’t need any queues to be running during migration. I’ll read that guide. Thanks!

And @adamzapasnik Application.fetch_env! pulls from the application map specified by the files in config/*.exs which are all populated at compile time even if they use System.get_env. I decided to go with Vapor using this Vapor config guide.

7

We have misunderstood each other. Sorry for that, I blame my phone…

What you could have done instead:
application.ex

defmodule MyApp.Application do
  def start(_, _) do
   ...
   {Oban, oban_opts}
  end

  defp oban_opts
    perform_jobs = System.get_env("PERFORM_JOB", true)
    opts =  Application.get_env(:my_app,Oban)
    if perform_jobs do
      opts
    else
      Keyword.update(opts, :queues, false)
    end
  end
end

What I was trying to point out was that this was evaluated at compile time:

@my_env_var Application.fetch_env!(:myapp, :some_var)

which should have been instead and it would have worked:

if Application.fetch_env!(:myapp, :some_var) do
   Oban.resume_queue(queue: :my queue)
end

I still would look into not booting up otp app while running just migrations.

1 Like
8

@adamzapasnik I agree, we should only start what is necessary. And thanks for the tip, you showed the same thing that @sorentwo shows in his guide. It’s a handy way to update configuration at application startup.

1 Like