Ecto.Adapters.SQL.Sandbox times out in CI (GH Actions/Semaphore)

Hi,

I’m looking for advice on setting up E2E testing with Cypress as a part of the CI/CD pipeline. The suite works perfectly fine, but it’s a total trainwreck with Github Actions and Semaphore.

I hope I’m just missing a timeout setting somewhere.

The whole approach is a somewhat low-key solution:

  • clone the e2e suite alongside the main app
  • start phx.server in the background
  • launch Cypress

Cypress’s capability to start a system under tests makes life a bit easier.

As I wrote, the problem is that Phoenix is starting to drop DB connections.

Here is the snippet from Semaphore:

1:32:32.325 [error] #PID<0.912.0> running MimisbrunnrWeb.Endpoint (connection #PID<0.805.0>, stream id 3
) terminated
Server: localhost:1080 (http)
Request: DELETE /sandbox
** (exit) an exception was raised:
    ** (File.Error) could not stream "apps/mimisbrunnr_web/lib/mimisbrunnr_web/endpoint.ex": no such file
 or directory
        (elixir 1.13.4) lib/file/stream.ex:83: anonymous fn/3 in Enumerable.File.Stream.reduce/3
        (elixir 1.13.4) lib/stream.ex:1517: anonymous fn/5 in Stream.resource/3
        (elixir 1.13.4) lib/stream.ex:1719: Enumerable.Stream.do_each/4
        (elixir 1.13.4) lib/enum.ex:4144: Enum.split/2
        (plug 1.13.6) lib/plug/debugger.ex:472: Plug.Debugger.get_snippet/2
        (plug 1.13.6) lib/plug/debugger.ex:328: Plug.Debugger.each_frame/4
        (elixir 1.13.4) lib/enum.ex:1715: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
        (elixir 1.13.4) lib/enum.ex:1715: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3

Depending on the debug settings I can also get

Error:  GenServer #PID<0.627.0> terminating
31484
** (MatchError) no match of right hand side value: :not_found
31485
    (phoenix_ecto 4.4.0) lib/phoenix_ecto/sql/sandbox_session.ex:22: Phoenix.Ecto.SQL.SandboxSession.handle_call/3

To isloate tests I followed guide from Dockyard https://dockyard.com/blog/2017/11/15/how-to-add-concurrent-transactional-end-to-end-tests-in-a-phoenix-powered-ember-app.

config/acceptance_tests.exs

import Config

config :mimisbrunnr, sql_sandbox: true

config :mimisbrunnr, Mimisbrunnr.Repo,
  pool: Ecto.Adapters.SQL.Sandbox,
  pool_size: 10

endpoint.ex:

  if Application.compile_env(:mimisbrunnr, :sql_sandbox) do
    plug Phoenix.Ecto.SQL.Sandbox,
      at: "/sandbox",
      header: "x-user-agent",
      repo: Mimisbrunnr.Repo,
      timeout: 30_000 # default
  end

If i bump timeout to 150_000 I’m getting

error] GenServer #PID<0.1276.0> terminating
** (MatchError) no match of right hand side value: :not_found
    (phoenix_ecto 4.4.0) lib/phoenix_ecto/sql/sandbox_session.ex:27: Phoenix.Ecto.SQL.SandboxSession.handle_info/2
    (stdlib 3.17.2) gen_server.erl:695: :gen_server.try_dispatch/4
    (stdlib 3.17.2) gen_server.erl:771: :gen_server.handle_msg/6
    (stdlib 3.17.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message: :timeout
State: %{client: #PID<0.1275.0>, repo: Mimisbrunnr.Repo, sandbox: Ecto.Adapters.SQL.Sandbox}
[error] Postgrex.Protocol (#PID<0.476.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.1416.0> exited
[error] Postgrex.Protocol (#PID<0.474.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.1415.0> exited
[error] Postgrex.Protocol (#PID<0.473.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.1418.0> exited
[error] Postgrex.Protocol (#PID<0.481.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.1419.0> exited
[error] Postgrex.Protocol (#PID<0.479.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.1442.0> exited
[error] Postgrex.Protocol (#PID<0.478.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.1445.0> exited

If the app doesn’t checkout session, then everything is fine. That is I comment out those two blocks:

Before(async () => {
  await cy.request('POST', sandboxUrl).then((response) => {
    context.sandboxSessionId = response.body;
    return context;
  });
});

After(() => {
  cy.request({
    method: 'DELETE',
    url: sandboxUrl,
    headers: { 'x-user-agent': context.sandboxSessionId },
    retryOnStatusCodeFailure: false,
    failOnStatusCode: false,
  });
});

Did anyone rune into this problem?

Did you have any luck reproducing it locally?

Without having access to more info, I am going to guess that your test being concurrent would not be compatible with how Ecto sandboxes by default in tests.

You can consider shared mode, cf. Ecto.Adapters.SQL.Sandbox — Ecto SQL v3.8.2