Testing in Ash & "(DBConnection.OwnershipError) cannot find ownership process for #PID"

[zac]

I just spent half a day trying to figure out how to get around this:

(DBConnection.OwnershipError) cannot find ownership process for #PIDxxx

Most of the time was spent trying to find decent references. I didn’t find anything in the Ash docs itself, and the examples I did find made no mention of connection contention (in fact, the examples are very straightforward using ETS, so contention probably isn’t an issue…)

I’m thinking an edit to the Ash Testing topic would be nice…? Perhaps adding mention of how this can seemingly bite you out of nowhere (my simple app was fine, then “something happened,” it tipped over the edge I guess, and suddenly all my tests stopped working).

I’d be happy to do a PR. Haven’t done one before, imagine the docs in github somewhere…

All that said, before making said PR, wanted to double-check what I’ve learned, as this was new territory for me:

(DBConnection.OwnershipError) cannot find ownership process for pid … root cause

Ecto docs and a few posts quickly point toward shared connection contention, particularly when using async tests.

What confused me: Various posts / Ecto docs made it sound like turning off async: true would fix the issue. Not so that did nothing.

Other info led me to believe switching from a :manual to either :auto (nope) or :shared (doesn’t exist anymore) sandbox mode would help. (Neither did).

While both strategies are offered in the Ecto docs, neither did anything toward fixing the problem. ¯_(ツ)_/¯

Ecto docs are confusing

Just that. The Ecto.Adapters.SQL.Sandbox docs offer a few different strategies. The way I interpreted the docs, I should be adding this to my tests:

    setup do
      # Explicitly get a connection before each test
      :ok = Ecto.Adapters.SQL.Sandbox.checkout(WasteWalk.Repo)
      # Setting the shared mode must be done only after checkout
      Ecto.Adapters.SQL.Sandbox.mode(WasteWalk.Repo, {:shared, self()})
    end

That did fix the tests but left me with another problem. All kinds of Postgrex.Protocol (#PID<0.404.0>) disconnected: ** (DBConnection.ConnectionError) cropping up (not just in my tests but a lot of framework tests). Made the situation worse.

What I settled on…

I ended up with just this in my tests:

    setup do
      :ok = Ecto.Adapters.SQL.Sandbox.checkout(WasteWalk.Repo)
    end

This is a new project – major upgrade from my last one (which used older Elixir, Phoenix, Ash). Didn’t have to worry about any of this “in the old days,” so… something new.

So far so good. Tests are working, nothing else has broken. I still don’t completely understand why it’s necessary (well, in theory, I suppose I do… more accurately, I don’t understand why turning on async:true didn’t solve the problem, nor why using a shared connection resulted in so much horrible fallout).

Anyhow, any further advice or explanation is welcome. Like I said, if there’s consensus this would be a good add for the docs I’ll do a PR. Hopefully with a better explanation than, “well, this fixed it, dunno why tho.”

[zac]

Gah.

Ok, saga not over.

I write doctests. Sometimes, not always. But apparently this is also causing intermittent issues stemming from the same problem. For instance:

  @moduledoc """
    Blah blah blah... nice docs...
      iex> {:ok, stereotype} = WasteWalk.Sessions.Stereotype.new("Engineering", :value_add)
      ...

But alas, when I try to run the doctests:

defmodule WasteWalk.Sessions.TestStereotype do
  use ExUnit.Case, async: true
  require Ash.Query

  doctest WasteWalk.Sessions.Stereotype
  ...

I’m back to those DBConnection.OwnershipError problem. Anyone know how to make doctests work in Ash?

  1) doctest module WasteWalk.Sessions.Stereotype (1) (WasteWalk.Sessions.TestStereotype)
     test/waste_walk/sessions/stereotype_test.exs:7
     match (=) failed
     code:  {:ok, stereotype} = WasteWalk.Sessions.Stereotype.new("Engineering", :value_add)
     left:  {:ok, stereotype}
     right: {
              :error,
              %Ash.Error.Unknown{bread_crumbs: ["Error returned from: WasteWalk.Sessions.Stereotype.create"],  changeset: "#Changeset<>",  errors: [%Ash.Error.Unknown.UnknownError{error: "** (DBConnection.OwnershipError) cannot find ownership process for #PID<0.508.0>.\n\nWhen using ownership, you must manage connections in one\nof the four ways:\n\n* By explicitly checking out a connection\n* By explicitly allowing a spawned process\n* By running the pool in shared mode\n* By using :caller option with allowed process\n\nThe first two options require every new process to explicitly\ncheck a connection out or be allowed by calling checkout or\nallow respectively.\n\nThe third option requires a {:shared, pid} mode to be set.\nIf using shared mode in tests, make sure your tests are not\nasync.\n\nThe fourth option requires [caller: pid] to be used when\nchecking out a connection from the pool. The caller process\nshould already be allowed on a connection.\n\nIf you are reading this error, it means you have not done one\nof the steps above or that the owner process has crashed.\n\nSee Ecto.Adapters.SQL.Sandbox docs for more information.", field: nil, value: nil, splode: Ash.Error, bread_crumbs: ["Error returned from: WasteWalk.Sessions.Stereotype.create"], vars: [], path: [], stacktrace: #Splode.Stacktrace<>, class: :unknown}]}
            }
     stacktrace:
       (for doctest at) lib/waste_walk/sessions/stereotype.ex:15: (test)

Also would be good material for a docs update PR…

[zachdaniel]

I think this would pretty much always have been necessary. I don’t remember a time where the SQL sandbox wasn’t a consideration. It’s not something new.

Have you set the disable async option described here? Testing — ash v3.5.33

How did you generate this application?

EDIT: I don’t believe this issue has anything at all to do with contention or too many tests

[zachdaniel]

Perhaps taking a look at how the book’s example app does testing setup would help? GitHub - sevenseacat/tunez at end-of-chapter-10

[zac]

I think this would pretty much always have been necessary.

I don’t believe so. I generated this app, then copied over a lot of testing template code from an older project. The older project didn’t have any references to ...Sandbox.checkout() anywhere. And it was a pretty large project, did a lot of DB centric testing. But it’s also several years old. Maybe there was some other way of establishing the same safeguards buried in there somewhere…

Have you set the disable async option described here? Testing — ash v3.5.33

I did in fact have both:

config :ash, :disable_async?, true
config :ash, :missed_notifications, :ignore

In my test.exs for some time. At the moment, I have it commented out and all seems to be working, along with a note that I might want to uncomment it again… Neither of these seem to have a bearing on the DBConnection issue.

For some reason I believe that async testing can expose more issues. Well, I suppose it does, but maybe it’s unnecessary.

How did you generate this application?

From here, w/ the latest distros (I haven’t done much beyond set up a basic schema, half a dozen resources, and a little test framework… so it’s pretty generic “out of the box” still):

mix igniter.new waste_walk --install ash,ash_postgres,ash_phoenix --with phx.new

I took a look at the book link… and I’m noticing that you do not seem to have:

setup do
  # Checkout connection to avoid issues with DBConnection.OwnershipError:
  :ok = Ecto.Adapters.SQL.Sandbox.checkout(WasteWalk.Repo)
end

Did I miss it? Which is interesting, because if I don’t put that on my tests, it all goes to hell. Specifically, the DBConnection error referenced in the title.

For continuity here’s my current test file. Note the setup section. If I take that out, it craps out. Likewise, the doctests will fail if I try to use them.

defmodule WasteWalk.Sessions.TestStereotype do
  use ExUnit.Case, async: true
  require Ash.Query

  # THIS blows up no matter what (DBConnection.OwnershipErrors, can't find a workaround):
  # doctest WasteWalk.Sessions.Stereotype

  describe "stereotypes must" do
    setup do
      # Checkout connection to avoid issues with DBConnection.OwnershipError:
      :ok = Ecto.Adapters.SQL.Sandbox.checkout(WasteWalk.Repo)
    end

    test "be persistent, and require a name" do
      {:ok, _} = WasteWalk.Sessions.Stereotype.new("Engineering", nil)
      {:error, %Ash.Error.Invalid{}} = WasteWalk.Sessions.Stereotype.new("", nil)
      {:error, %Ash.Error.Invalid{}} = WasteWalk.Sessions.Stereotype.new(nil, nil)
    end

    test "have a type" do
      {:ok, _} = WasteWalk.Sessions.Stereotype.new("Engineering", :value_add)
      {:ok, _} = WasteWalk.Sessions.Stereotype.new("Random meetings", :non_value_add)
      {:error, %Ash.Error.Invalid{}} = WasteWalk.Sessions.Stereotype.new("Engineering", :unknown_type)
    end

    test "not allow duplicates" do
      {:ok, _} = WasteWalk.Sessions.Stereotype.new("Engineering", nil)
      {:error, %Ash.Error.Invalid{}} = WasteWalk.Sessions.Stereotype.new("Engineering", nil)
    end
  end
end

I haven’t added in any authorization yet, although it’s in the base project. Commented out on this resource though. Thought I’d get the basics working before I tackled learning authorization…

[zac]

Just to double check, I uncommented the disable_async? bits… no difference.

As an FYI, the generators did not include either of them… had to put them in myself. That said, having done so, I didn’t notice any change in performance.

config :ash, :disable_async?, true
config :ash, :missed_notifications, :ignore

[zachdaniel]

Okay, found an issue where the installer would sometimes not set those when installing into an existing app.

Typically, you see something like use MyApp.DataCase, not use ExUnit.Case

For tunez that looks like this, which has sandbox setup.

defmodule Tunez.DataCase do
  @moduledoc """
  This module defines the setup for tests requiring
  access to the application's data layer.

  You may define functions here to be used as helpers in
  your tests.

  Finally, if the test case interacts with the database,
  we enable the SQL sandbox, so changes done to the database
  are reverted at the end of every test. If you are using
  PostgreSQL, you can even run database tests asynchronously
  by setting `use Tunez.DataCase, async: true`, although
  this option is not recommended for other databases.
  """

  use ExUnit.CaseTemplate

  using do
    quote do
      alias Tunez.Repo

      import Ecto
      import Ecto.Changeset
      import Ecto.Query
      import Ash.Test
      import Tunez.DataCase
      import Tunez.Generator
      import Tunez.Support.Helpers
    end
  end

  setup tags do
    Tunez.DataCase.setup_sandbox(tags)
    :ok
  end

  @doc """
  Sets up the sandbox based on the test tags.
  """
  def setup_sandbox(tags) do
    pid = Ecto.Adapters.SQL.Sandbox.start_owner!(Tunez.Repo, shared: not tags[:async])
    on_exit(fn -> Ecto.Adapters.SQL.Sandbox.stop_owner(pid) end)
  end

  @doc """
  A helper that transforms changeset errors into a map of messages.

      assert {:error, changeset} = Accounts.create_user(%{password: "short"})
      assert "password is too short" in errors_on(changeset).password
      assert %{password: ["password is too short"]} = errors_on(changeset)

  """
  def errors_on(changeset) do
    Ecto.Changeset.traverse_errors(changeset, fn {message, opts} ->
      Regex.replace(~r"%{(\w+)}", message, fn _, key ->
        opts |> Keyword.get(String.to_existing_atom(key), key) |> to_string()
      end)
    end)
  end
end

[zac]

Aha. I see.

I went back to look at the Ash Test Resources how-to-guide. It’s simplified (ETS) but, only shows how to use ExUnit.Case:

defmodule ActionInvocationTest do
  use ExUnit.Case
  ...

And then there’s the section you pointed to in Ash Testing, which talks about about turning off the async? options.

BUT… nowhere have I found mention of using DataCase.

It’s probably in there, but it’s not on those two central documents about testing. Might make sense to add it?

Switching from ExUnit.Case over to WasteWalk.DataCase solved the problem across the board. Got the doctests working too.

[zachdaniel]

I think this is an example of something that Ecto configures and we just sort of sit on top of. Please open an issue on ash to enhance the docs on this and I will get to it when I can.