Tests that use randomness in a module property, that randomness is not seeded

Chat / Discussions
1

This is a public service announcement, rather than a question.

We had a flappy test that did not fail again when using the same test seed. The test had no processes / async in it; it was so confusing and rare that we just ignored it for months. We’ve finally tracked it down to a use of Enum.random in a property of the test module, eg.

defmodule MyApp.YourTest do
  @item_1_params %{
    id: Faker.UUID.v4(),
    quantity: Enum.random(1..10)
  }
end

The above randomness is not seeded with the test seed. It now makes sense, the compilation of the tests must be a different phase of the test process from running tests, with seeding happening in between.