Temper - Flaky test detection for ExUnit

Hi everyone!

I’d like to introduce Temper, a library that finds flaky tests in ExUnit suites: the tests that pass and fail without the code changing.

Before building it I searched the forum and found surprisingly little on the topic, even though it bites almost every team eventually. There are brief discussions that touch on it (usually framed as a retry problem, or as a one-off hunt for a specific race condition), but nothing that really delves into it. Temper starts from a different premise: a flaky test is telling you something about a race condition, shared state leaking between tests, or a timing assumption, and retrying it into silence hides the message. Temper’s job is to make the message visible: which tests flake, how often, and under which seeds, so you can fix the brittleness instead of ignoring it.

How it works

Temper is a passive ExUnit formatter: it records every test outcome to a local history file (with git SHA, seed, partition, and CI metadata) and reports tests with divergent outcomes on the same git SHA. The code didn’t change, but the result did. No retries, no re-runs, zero added test time. Every mix test run you were already doing becomes one observation, and evidence accumulates across runs.

Setup is two lines:

# mix.exs
{:temper, "~> 0.2", only: [:dev, :test], runtime: false}

# test/test_helper.exs
ExUnit.start(formatters: [ExUnit.CLIFormatter, Temper.Formatter])

Then, once history accumulates:

$ mix temper.report
Flaky tests (divergent outcomes on same git SHA):

  MyApp.UserTest test creates user with valid attrs
    test/my_app/user_test.exs:42  async: true
    12 runs on a1b2c3d: 10 passed / 2 failed (16.7% flake rate)
    failing seeds: 493821, 110394

A few design choices I care about:

• Zero false positives over recall. A test that fails on one commit and passes on the next is a fix (or a break), not a flake, so detection is same-SHA only. Divergence involving dirty-working-tree runs is reported separately as a “suspect” with lower confidence. The goal is a report you can trust over one that cries wolf.
• It never breaks your suite. If Temper itself hits an error, it warns once and goes inert for the rest of the run.
• CI-aware. It detects GitHub Actions / GitLab CI / CircleCI automatically, and the README documents persisting history across CI runs (including partitioned/parallel jobs and umbrella projects). mix temper.doctor diagnoses the setup problems that would otherwise fail silently.

Real catches from our own CI

We’ve been dogfooding Temper on a production umbrella app, and it’s already surfacing real flakes (module names anonymized):

Flaky tests (divergent outcomes on same git SHA):
  my_app — 1 flaky across 1 files
    MyApp.Podcasts.EpisodeIngestionTest test suppresses re-ingested episode notifications for 24 hours
      apps/my_app/test/my_app/podcasts/episode_ingestion_test.exs:53  async: false
      2 runs on 1581e30: 1 passed / 1 failed (50.0% flake rate)
      failing seeds: 772157

Flaky tests (divergent outcomes on same git SHA):
  my_app — 1 flaky across 1 files
    MyApp.Catalog.CacheWarmerTest test warm handle_info :refresh warms the cache
      apps/my_app/test/my_app/catalog/cache_warmer_test.exs:57  async: false
      2 runs on 1bc608a: 1 passed / 1 failed (50.0% flake rate)
      failing seeds: 836699

Both are classic flake archetypes (one leans on wall-clock time, the other asserts on a GenServer’s async work), and both had been quietly passing-and-failing in CI without anyone tracking them.

Where this is going

Detection is the foundation, but the larger goal is to make dealing with flaky tests active and educational, rather than “retry and forget”. Without promising dates, the direction is:

• A knowledge base of flakiness types: the common causes (timing, shared state, ordering, external services…) with explanations and concrete fixes, so a report doesn’t just say what flaked but helps you understand why.
• A CI-integrated service that ingests test history and comments on PRs when a flake is detected.
• An incident dashboard to track flakes over time: when they appeared, how often they bite, when they were fixed.

The philosophy stays the same throughout: detection first, trust before automation. Temper will never retry, quarantine, or block your CI behind your back.

Feedback welcome!

Temper is young (pre-1.0), and early feedback is what shapes what gets built next. I’d love for you to try it on your suite and tell me what happens: a test wrongly flagged, one that should have been, a confusing report, anything at all. Issues and PRs are very welcome, and if you find it useful, a star on GitHub helps a lot.

Docs: temper v0.2.1 — Documentation
Hex: temper | Hex