Is it possible to create a test semaphore?

Questions / Help
,
1

Hi!

I’ve been having some difficulties in integrating a part of a project into Elixir.

Namely I want to create a sort of test-semaphore which only runs specific tests when explicitly told so.

The mechanism behind this is that there’s an active WebSocket connection connected to a phoenix server which enqueues tests and once the server responds with A-OK on a specific test the test should only then be ran.

I know that I could wrap some stuff around in a new mix task and then just alias that as mix task and make it so that mix run test_path:line is started on every response but this would greatly increase the test runtime as it would need to restart some potentially long-starting services on every test unit.

I tried fiddling around with test_helper.exs and modified ExUnit.start() to have autorun: false and include: [] but that still started ALL tests which I don’t really want, I want to be able to start specific test-suites on demand.

Is there any better approach to do this than to make a wrapper which runs mix test file:line every time a specific test needs to be ran? Kinda stuck to a wall and I’d prefer not to make an XY problem.

2

Tbh I’d start by not attempting to reuse any of the pieces integrated in mix. Unless you’ve dug into their code I would imagine it being quite hard to figure out if they support your usecase or not. I’d build a custom mix task and integrate it only with the ExUnit api, where you know exactly what is called or not.

3

Hm. While I do get it (creating a custom mix task which would control the dataflow), I would probably prefer to be able to do this in test_helper.exs as, I assume, otherwise it would create an issue for other, more complex, test helpers potentially not being ran.

4

I’ve gotten to this point:

defmodule Mix.Tasks.App.Test do
  require Logger
  use Mix.Task
  
  def run(_args) do    
    Application.ensure_all_started(:ex_unit)

    __MODULE__.Traverser.recursive_ls("test")
    |> Enum.filter(&Regex.run(~r/(_test.exs|_helper.exs)$/, &1))
    |> tap(&Logger.debug("Matched tests and helpers are: #{inspect(&1)}"))
    |> Enum.each(&Code.require_file/1)
    
    time = ExUnit.Server.modules_loaded(false)
    options = ExUnit.configuration()
    
    # ExUnit.Runner.run(options, time)
    # |> IO.inspect()
  end
  
  defmodule Traverser do
    def recursive_ls(path) do
      case File.ls(path) do
        {:ok, entries} ->
          entries
          |> Enum.map(&Path.join(path, &1))
          |> Enum.flat_map(&process_entry/1)
      end
    end
    
    defp process_entry(entry) do
      if File.dir?(entry) do
        recursive_ls(entry)
      else
        [entry]
      end
    end
  end
end

Where the modules ARE loaded and I can run some tests when I uncomment the commented code.

I was able to prevent ExUnit from starting via a Application variable due to helper modules, with which I’m fine to setup:

$ cat config/test.exs
import Config

config :ex_unit,
  autorun: false

Now I guess it to see how to:

  • Get the list of possible tests without running them
  • Be able to run on a per-test basis with the top-level ExUnit API

If I can’t do that, I’ll have to either work my way around private API calls or reimplement the functionality (which i really don’t want to do) which seems possible with ExUnit.Runner.

5

So at least some progress, I was able to not start all tests, and then start them via another node, to simulate another process starting them within the app:

The running node:

iex --name runner@192.168.1.130 --cookie my_secret_password -S mix app.test

  def run(_args) do    
    Application.ensure_all_started(:ex_unit)

    __MODULE__.Traverser.recursive_ls("test")
    |> Enum.filter(&Regex.run(~r/(_test.exs|_helper.exs)$/, &1))
    |> Enum.each(&Code.require_file/1)

    
    # The take calls fail unless this is called beforehand
    _time = ExUnit.Server.modules_loaded(false)
    
    async =
      ExUnit.Server.take_async_modules(5_000)
      |> IO.inspect(label: "async")
    
    sync =
      ExUnit.Server.take_sync_modules()
      |> IO.inspect(label: "sync")
  end

The control node on a different computer:

iex --name control@192.168.1.2 -S mix

iex(control@192.168.1.2)1> Node.connect(:"runner@192.168.1.130")
true

iex(control@192.168.1.2)2> Node.spawn_link(:"runner@192.168.1.130", fn -> ExUnit.Server.add_sync_module(AppTest) end)
#PID<13285.195.0>

iex(control@192.168.1.2)3> Node.spawn_link(:"runner@192.168.1.130", fn -> ExUnit.Runner.run(ExUnit.configuration(), ExUnit.Server.modules_loaded(false)) end)
#PID<13285.196.0>
...............
Finished in 85.7 seconds (79.2s on load, 0.00s async, 6.5s sync)
15 tests, 0 failures

Randomized with seed 476053

Although there might be an issue with concurrency like this.

Worst-case scenario, I can make multiple nodes as runners via the mix task and then connect to each one of them. That way I’d have multiple instances of the application and work around concurrency that way if there’s no alternative.

6

So it was just a fried brain, basically the way I wrote the above elixir code works essentially as the semaphore I need and there’s no need to worry about concurrency as it’s always going to be one thread which does those tests.

BUT if I were to have something which has multiple threads I’d use multiple local nodes and connect them. Why? Because the ExUnit.Server uses the module as the GenServer name, so any form of concurrency with isolated threads/data would be borderline impossible.