Debugging obscure process crashes?

Your process receives and :EXIT message from another process that is linked to the former.

Do you start a process from your job process with a SomeModule.start_link or spawn_link call? The process with pid <0.13246.0> was killed and as it was linked to your own process, your process exited with that same reason.

You may create a test and directly call your Job.perform function repeatedly, without starting oban. This will be easier to get a proper stacktrace.

I’m throwing a dart in the dark here, but I’ve been bitte by HttPoison/hackney pools before, where if the request crashes it doesn’t release the connection, starving the pool in a short time.

Try setting pool: false and see if that helps maybe?

Hey, thanks for the idea. I never thought of this – this is much easier to test in isolation.

I’ve performed a simple load test, and everything I can throw at it seems to work. The job executes well. I’ll continue testing the worker in isolation and report back – I’m hoping to get a better stack trace than whan Oban provides.

Enum.each(1..100, fn _ -> Parcel.Jobs.DeliverWebhook.perform(job) end)

I figured it could be caused by HTTPoison pools – will absolutely try this, as I have no clue where the process is getting sent a :killed.

Just found that pools limit the amount of concurrent connections you can execute, will disable them. This could be the problem: too many Oban jobs are executing in parallel and the pool gets killed?

normally you would wait for a connection to become available, but when you have the issue of having the connections not being released back into the pool you basically loose capacity silently until things blow up.

The worst thing is that since it’s very easy to just leave the :default pool, you can have totally unrelated parts of your app (that also use the default pool) causing this issue.

Your process receives and :EXIT message from another process that is linked to the former.

This kind of thing could indeed be the root cause of pool starvation, to the point where you have your jobs waiting forever for a connection that is never going to be put back into the pool

I’ve disabled the connection pool and wrote a small testing Mix script to load test the behavior. I’m seeing no :killed yet, but will continue testing over the following days under many conditions (including starving the system of memory).

Reporting back in a bit.

  @impl Mix.Task
  def run(_) do
    {:ok, pid} = Task.Supervisor.start_link()

    Enum.map(0..100, fn _ ->
      Task.Supervisor.async(pid, fn ->
        HTTPoison.start()

        req =
          HTTPoison.post!(
            "http://localhost:4000/v1/events/",
            Jason.encode!(%{
              subscriber_id: "sub_0000",
              name: "test.event",
              payload: %{
                hello: "world"
              }
            }),
            [{"Authorization", "Bearer key_0000"}, {"Content-Type", "application/json"}]
          )

        Logger.debug("Sent request, received #{req.status_code}")
      end)
    end)
    |> Task.await_many()
  end

I’ll continue testing over the next couple of days. I got some errors, but because the system ran out of file descriptors (I bombed the system with requests). I also saw a Killed error somewhere, so I suspect the problem persists.

I will continue testing in different system conditions, such as high memory usage and long uptime (lots of hot reloads from Phoenix, for example) and report back. I’m still wondering what caused the problem, or if I can prove the connection pooling caused it definitively.

I think I’ve seen the problem more with long-running instances of mix phx.server, which include closing the laptop lots and leaving the process on for days on end.

config :logger,
  handle_otp_reports: true,
  handle_sasl_reports: true

You’ll probably see the other error with those on.

This is not concurrent. Your other snippet with Task.Supervisor.async is the way to go. Now, it can be actually pretty fast. What I would do is to simulate load by adding a sleep(1 second) in the server that responds at http://localhost:4000/v1/events/.

You said that you do not see the problem in tests. Does the problem still exists when using Oban?

The problem is primarily when using Oban, yes.

I will add a Process.sleep at the controller and the Oban job to simulate load

EDIT: Okay, so what ends up happening under the load script is that the server locks up and requests don’t go through. The error persisted.

requests start locking up, nothing sends1200×246 38.6 KB

crashes everywhere1133×1233 285 KB

I’m so confused. I think I found how to replicate it. This only happens when the laptop goes to sleep and comes back. I restarted the process and it’s processing jobs, handling requests like normal without Phoenix locking up.

Odd state bug? I’m worried this might end up happening in production with long-running instances.