Temper - Flaky test detection for ExUnit

volcov

Hi everyone!

I’d like to introduce Temper, a library that finds flaky tests in ExUnit suites: the tests that pass and fail without the code changing.

Before building it I searched the forum and found surprisingly little on the topic, even though it bites almost every team eventually. There are brief discussions that touch on it (usually framed as a retry problem, or as a one-off hunt for a specific race condition), but nothing that really delves into it. Temper starts from a different premise: a flaky test is telling you something about a race condition, shared state leaking between tests, or a timing assumption, and retrying it into silence hides the message. Temper’s job is to make the message visible: which tests flake, how often, and under which seeds, so you can fix the brittleness instead of ignoring it.

How it works

Temper is a passive ExUnit formatter: it records every test outcome to a local history file (with git SHA, seed, partition, and CI metadata) and reports tests with divergent outcomes on the same git SHA. The code didn’t change, but the result did. No retries, no re-runs, zero added test time. Every mix test run you were already doing becomes one observation, and evidence accumulates across runs.

Setup is two lines:

# mix.exs
{:temper, "~> 0.2", only: [:dev, :test], runtime: false}

# test/test_helper.exs
ExUnit.start(formatters: [ExUnit.CLIFormatter, Temper.Formatter])

Then, once history accumulates:

$ mix temper.report
Flaky tests (divergent outcomes on same git SHA):

  MyApp.UserTest test creates user with valid attrs
    test/my_app/user_test.exs:42  async: true
    12 runs on a1b2c3d: 10 passed / 2 failed (16.7% flake rate)
    failing seeds: 493821, 110394

A few design choices I care about:

Zero false positives over recall. A test that fails on one commit and passes on the next is a fix (or a break), not a flake, so detection is same-SHA only. Divergence involving dirty-working-tree runs is reported separately as a “suspect” with lower confidence. The goal is a report you can trust over one that cries wolf.
It never breaks your suite. If Temper itself hits an error, it warns once and goes inert for the rest of the run.
CI-aware. It detects GitHub Actions / GitLab CI / CircleCI automatically, and the README documents persisting history across CI runs (including partitioned/parallel jobs and umbrella projects). mix temper.doctor diagnoses the setup problems that would otherwise fail silently.

Real catches from our own CI

We’ve been dogfooding Temper on a production umbrella app, and it’s already surfacing real flakes (module names anonymized):

Flaky tests (divergent outcomes on same git SHA):
  my_app — 1 flaky across 1 files
    MyApp.Podcasts.EpisodeIngestionTest test suppresses re-ingested episode notifications for 24 hours
      apps/my_app/test/my_app/podcasts/episode_ingestion_test.exs:53  async: false
      2 runs on 1581e30: 1 passed / 1 failed (50.0% flake rate)
      failing seeds: 772157

Flaky tests (divergent outcomes on same git SHA):
  my_app — 1 flaky across 1 files
    MyApp.Catalog.CacheWarmerTest test warm handle_info :refresh warms the cache
      apps/my_app/test/my_app/catalog/cache_warmer_test.exs:57  async: false
      2 runs on 1bc608a: 1 passed / 1 failed (50.0% flake rate)
      failing seeds: 836699

Both are classic flake archetypes (one leans on wall-clock time, the other asserts on a GenServer’s async work), and both had been quietly passing-and-failing in CI without anyone tracking them.

Where this is going

Detection is the foundation, but the larger goal is to make dealing with flaky tests active and educational, rather than “retry and forget”. Without promising dates, the direction is:

A knowledge base of flakiness types: the common causes (timing, shared state, ordering, external services…) with explanations and concrete fixes, so a report doesn’t just say what flaked but helps you understand why.
A CI-integrated service that ingests test history and comments on PRs when a flake is detected.
An incident dashboard to track flakes over time: when they appeared, how often they bite, when they were fixed.

The philosophy stays the same throughout: detection first, trust before automation. Temper will never retry, quarantine, or block your CI behind your back.

Feedback welcome!

Temper is young (pre-1.0), and early feedback is what shapes what gets built next. I’d love for you to try it on your suite and tell me what happens: a test wrongly flagged, one that should have been, a confusing report, anything at all. Issues and PRs are very welcome, and if you find it useful, a star on GitHub helps a lot.

Docs: temper v0.2.1 — Documentation
Hex: temper | Hex

9 comments

#library #temper #flaky

24 979 9

2026-09-02 12:10:32 UTC

Showing Posts 1 to 9

Show Best Posts
Show All (oldest first)
Show All (newest first)

dimitarvp

Love the name. It’s spot-on.

Post #1

frerich

Interesting! Since it correlates test executions based on Git SHA, I understand it’ll only flag a warning if you actually run a test suite repeatedly for the exact same Git commit?

I wonder how often that happens: typically, CI runs will be triggered due to repo activity - i.e. the Git SHA is most likely different, right?

Post #2

mudasobwa

Creator of Cure

That looks very promising.

Could you please clarify your workflow in a case the flake has been detected in CI. .temper directory is read-only in CI, right? So what happened when some test has been flaken out (pun intended)?

The CI run has some report, which is AFAIU kinda fire-and-forget, correct? Where the history comes from then? From local runs? The developer shoud be responsible enough to somehow add the result into .temper folder?

I mean, I would love to use something like this if it kept track on flakes on its own. I never saw a flake in local, they always happen to appear in CI, and I usually don’t need a tool to detect those, if an unrelated test fails out of the blue, it’s flaky, that’s it. And when it happens I usually have neither time nor willingness to deal with that mess.

On the other hand, it would be extremely helpful to collect flakes to be able to turn back to them once in a while and kill some. That said, my main question would be how do I preserve the history over CI runs in a human-readable format at hand?

Post #3

volcov

Thanks for the great question =)
You read it exactly right: a confirmed flake requires divergence on the same SHA. That’s deliberate, fail-on-A, pass-on-B is indistinguishable from a fix or a regression, and I’d rather miss a flake than cry wolf. (Divergence involving dirty-working-tree runs is still recorded, but hedged as a lower-confidence “suspect”).

Same-SHA repetition happens more often than it first seems, though. The big one is the “re-run the red job” reflex: that retry is a same-SHA pair, and a fail → pass pair on one commit is the strongest flake evidence there is. In our own CI the history cache key includes the run attempt precisely so a re-triggered run inherits the failed attempt’s history. Local runs between commits on a clean tree count too.

And you’ve identified exactly where this is heading: two planned features manufacture same-SHA repetition on purpose. Burn-in runs new or changed tests N times in the PR pipeline (a single green run can’t reveal a flake, so new flaky tests currently merge unflagged), and retry-on-detect forces a same-SHA second observation when a failure occurs. Both feed the same history, and both are coming soon in v0.3

Post #4

volcov

Thank you for the kind words and for such a thorough set of questions!

.temper isn’t read-only in CI; persisting it is the intended setup. The README covers it (including partitioned/parallel jobs): GitHub - volcov/temper · GitHub. We run this on a private umbrella app: actions/cache/restore before the tests (restore-keys fall through to a branch prefix, so you always pick up the most recent prior run, including the previous attempt of a re-run), then actions/cache/save with if: always(), because saving on failure is the point, failures are half the divergence signal.

So the report isn’t fire-and-forget: the history (JSONL, one run per line, greppable) is the persistent thing, and the report is regenerated from it. We run mix temper.report as a final if: always() informational step, so every CI run prints the current flake list in the job log, that’s the human-readable format at hand, and you can run it locally against the same cached history whenever you feel like killing a few.

On “kept track on its own”: agreed, and that’s the direction. The JSONL schema and the --json report payload were designed as an ingestion protocol, and the next milestone is a CI upload path plus a service that aggregates history across branches and repos, so collection needs zero developer discipline. Your framing (“collect flakes to turn back to them once in a while”) is exactly the use case it’s built around =)

Post #5

volcov

@mudasobwa your question stuck with me:

how do I preserve the history over CI runs in a human-readable format at hand?

It ended up shaping the release I just published: Temper 0.3.0 is essentially the answer to it.

Getting the history into your hands. Have each CI job upload its .temper/ directory as a build artifact, then download the artifacts (from one run or several) and merge them into a single deduplicated file:

$ mix temper.merge --output .temper/history-0.jsonl "artifacts/**/history-*.jsonl"
$ mix temper.report

Byte-identical lines are written once, so overlapping cache restores and re-downloaded artifacts never double-count an outcome.

Human-readable, and yours to script against. The history was always JSON Lines (one self-contained object per test outcome: git SHA, seed, timing, failure signature). What’s new is that the format is now a documented public contract: the History Schema guide covers every field, and the schema is versioned, so anything you build on it won’t break silently. If mix temper.report doesn’t answer your particular question, the file is plain material for your own scripts

I have other ideas in mind that will be coming soon, I’m getting great feedback from those who have been using it.

Post #6

mudasobwa

Creator of Cure

Awesome!

Scripting against is exactly what I was after. Thanks for reading between the lines of my request and for the fast feedback.

Post #7

krasenyp

@volcov Very promising library, thank you for releasing it but don’t you think it’s a little disrespectful to answer with LLM generated text?

Post #8

volcov

Hi @krasenyp, how are you?
Thanks a lot for your comment about the library. I really hope it helps solve a problem I’d noticed in our pipeline, one I imagine others out there might be facing too.

Regarding your question: English isn’t my native language, so I use a tool to help me fix issues like subject-verb agreement or grammatical errors, as well as to adjust the tone. I feel it’s important to convey the message clearly and correctly in a public forum that might serve as documentation or a resource for others. I’m learning with every reply, and I notice the tool makes fewer and fewer adjustments to my text over time. Plus, once I have the result, it’s easier to just copy the corrected text than to retype it.

So, in my humble opinion, and please forgive me if I offended you in any way (don’t worry, I’m writing this reply and checking it via Google Translate, just as I used to), I don’t think it’s disrespectful. After all, I wrote the answers myself and simply used the tool to help avoid mistakes.

Thanks again for paying attention to the contribution I wanted to share, even though the way I answered bothered you.

Post #9

— All posts loaded —