Run umbrella project tests sequencially

Questions / Help
, , ,
1

Background

I have an umbrella project with several apps.
Some of these apps have runtime libraries as dependencies, some apps depend on others and so on.

My objective is, when on the root level of the project, to run all the tests for all the child apps using mix test.

Problem

When on the root level, running mix test will run the tests of all child apps. However this happens in a parallel way, from what I can see.

This is a problem because some dependencies between child apps are shared. Consequently some apps succeed, and others fail, thus breaking my CI/CD pipeline.

This is not a problem at all if I run the tests for each child separately, but alas, I want to be able to use mix test from the root level.

What I tried

So, my initial approach was create an alias that replaces mix test with my own command:

defp aliases do
    [
      test: ["cmd --app app1 mix test --color", "cmd --app app2 mix test --color"]
    ]
  end

This will work and will execute the tests of the child apps sequentially.

Questions

However this solution is hard-coded and not very flexible. If tomorrow I remove an app, the command will break. If I add a new one, the command won’t run its tests unless I remember to edit it.

I wonder if there is a better way of doing this that would be:

  • flexible instead of hard-coded
  • run the tests sequentially instead of parallel
2

So the proper solution would be removing any global state from your testsuites, which makes mix test fail as it does.

But also mix.exs is elixir – you can use code to define the alias:

test = 
  Mix.Project.apps_paths()
  |> Keyword.keys()
  |> Enum.map(fn app -> "cmd --app #{app} mix test --color" end)

[test: test]
1 Like
3

Unfortunately this won’t work, as Mix.Project.app_paths() is not resolved at compile time. So when I run the command, line 2 gets a nil instead of a keyword list and the whole thing crashes.

I have also tried Mix.Project.config(), which does work at compile time, but sadly it has no information about child apps.

4

Path.wildcard("apps/*) would likely do fine as well :sweat_smile:

1 Like
5 
defp aliases do
  child_tests =
      Path.wildcard("apps/*")
      |> Enum.map(&String.replace(&1, "apps/", ""))
      |> Enum.map(fn app -> "cmd --app #{app} mix test --color" end)

    [test: child_tests]
end

This did the trick. Thanks :smiley: