cjk

Ecto timeout errors when wrapping into cachex calls in tests

Hi there,

I use Cachex to cache very frequent queries:

@spec get_setting_for_forum(%Forum{}) :: %Setting{} | nil
def get_setting_for_forum(%Forum{} = forum) do
  Cachex.fetch!(:cforum, "settings/forums/#{forum.forum_id}", fn ->
    from(setting in Setting, where: setting.forum_id == ^forum.forum_id)
    |> Repo.one()
  end)
end

@spec get_setting_for_user(%User{}) :: %Setting{} | nil
def get_setting_for_user(%User{} = user) do
  Cachex.fetch!(:cforum, "settings/users/#{user.user_id}", fn ->
    from(setting in Setting, where: setting.user_id == ^user.user_id)
    |> Repo.one()
  end)
end

This function loads a settings object from the database. It works fine in production and dev code, but in tests I get this error:

** (Cachex.ExecutionError) connection not available and request was dropped from queue after 602ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information

I call the getter functions like this:

def load_relevant_settings(forum, user)

def load_relevant_settings(nil, nil) do
  [get_global_setting()]
  |> Enum.reject(&is_nil(&1))
end

def load_relevant_settings(%Forum{} = forum, nil) do
  [get_global_setting(), get_setting_for_forum(forum)]
  |> Enum.reject(&is_nil(&1))
end

def load_relevant_settings(nil, %User{} = user) do
  [get_global_setting(), get_setting_for_user(user)]
  |> Enum.reject(&is_nil(&1))
end

def load_relevant_settings(%Forum{} = forum, %User{} = user) do
  [get_global_setting(), get_setting_for_forum(forum), get_setting_for_user(user)]
  |> Enum.reject(&is_nil(&1))
end

I can’t figure out what’s going wrong. Has anybody an idea?

Edit: I already tried to set the pool size to a very high value (100), but it didn’t help.

Best regards,
CK

15 comments

#ecto #cachex

0 2879 15

2021-03-01 14:59:25 UTC

First Post!

cjk

I think I figured out what happens. The full stacktrace is as follows:

      (cachex) lib/cachex.ex:1407: Cachex.unwrap_unsafe/1
       (cforum) lib/cforum/config_manager.ex:278: Cforum.ConfigManager.conf/3
       (cforum) lib/cforum/config_manager.ex:262: Cforum.ConfigManager.conf/3
       (cforum) lib/cforum/forums/message_indexer_job.ex:21: anonymous fn/2 in Cforum.Forums.MessageIndexerJob.index_message/2
       (cforum) lib/cforum/forums/messages.ex:524: Cforum.Forums.Messages.index_message/2
       (cforum) lib/cforum/forums/threads.ex:431: Cforum.Forums.Threads.create_message/5
       (ecto_sql) lib/ecto/adapters/sql.ex:814: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
       (db_connection) lib/db_connection.ex:1349: DBConnection.run_transaction/4
       (cforum) lib/cforum/forums/threads.ex:403: Cforum.Forums.Threads.create_thread/5
       (cforum) lib/cforum_web/controllers/thread_controller.ex:139: CforumWeb.ThreadController.create_thread/2
       (cforum) lib/cforum_web/controllers/thread_controller.ex:1: CforumWeb.ThreadController.action/2
       (cforum) lib/cforum_web/controllers/thread_controller.ex:1: CforumWeb.ThreadController.phoenix_controller_pipeline/2
       (cforum) lib/cforum_web/endpoint.ex:1: CforumWeb.Endpoint.instrument/4
       (phoenix) lib/phoenix/router.ex:275: Phoenix.Router.__call__/1
       (cforum) lib/cforum_web/endpoint.ex:1: CforumWeb.Endpoint.plug_builder_call/2
       (cforum) lib/cforum_web/endpoint.ex:1: CforumWeb.Endpoint.call/2
       (phoenix) lib/phoenix/test/conn_test.ex:235: Phoenix.ConnTest.dispatch/5
       test/cforum_web/controllers/thread_controller_test.exs:92: (test)

The application is in the middle of a database operation. Since the Ecto.Adapters.SQL.Sandbox adapter wraps every test case in a giant transaction there is only one ecto process available, right? This would then lead to a deadlock when requesting multiple times for a database connection, wouldn’t it?

How can I circumvent this?

Edit: but if this is the case, why does everything work fine when not wrapping the getter functions in Cachex.fetch?

Best regards,
CK

Post #1

Most Liked

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

You can do:

def get_setting_for_forum(%Forum{} = forum) do
  caller = self()
  Cachex.fetch!(:cforum, "settings/forums/#{forum.forum_id}", fn ->
    from(setting in Setting, where: setting.forum_id == ^forum.forum_id)
    |> Repo.one(caller: caller)
  end)
end

Post #3

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Post #9

al2o3cr

One gotcha with this - AFAIK there’s no way to “un-allow” a connection, and Ecto gets grumpy if you call allow with a second PID. This makes for intermittently-failing tests when there’s a worker pool, as you’ll only get an error when a worker gets reused.

We ran into this issue with :poolboy; our fix was to have workers return :stop in tests so they don’t get re-used. This works, EXCEPT for very occasionally when there’s already a process waiting for a pool worker when the call returns; a fast-path optimization in Poolboy hands the now-exiting worker PID to the waiting process, which then fails