Book 10 · Testing & QualityPage 10 of 36

Flaky Test

Updated September 30, 20262 min read

In short

A flaky test is an automated test that sometimes passes and sometimes fails without any change to the code, which makes its results hard to trust.

What is a flaky test?

A flaky test gives different results on different runs even though neither the code nor the test has changed. It might pass nine times and fail on the tenth, which makes it hard to tell whether a failure points to a real bug or just bad luck.

Flakiness usually comes from nondeterminism, meaning something in the test is not the same on every run. Common causes include timing problems and race conditions in asynchronous code, fixed sleep calls that are sometimes too short, tests that depend on each other's leftover data or run order, real network calls, random values, and the current date or time zone. End-to-end tests are especially prone to flakiness because they involve browsers, networks, and many moving parts.

A flaky test is like a smoke alarm that goes off at random: after a few false alarms, people stop reacting, even when there is a real fire. The same happens to teams whose pipelines fail randomly, as developers learn to click rerun and real bugs slip through.

The fix is to find and remove the source of randomness: wait for a specific condition instead of a fixed time, reset data before each test, mock the clock and external services, and seed random number generators. Many teams quarantine a flaky test, moving it out of the required suite temporarily, while they investigate. Automatic retries can hide the symptom, but they don't fix the cause.

Key takeaways

A flaky test passes and fails randomly with no code change.
Common causes are timing issues, shared state, test order, network calls, and dates.
Flaky tests erode trust in the whole test suite.
Fix the root cause instead of relying on automatic retries.

Example

Fixing a timing-based flaky testjavascript

// Flaky: assumes the data always arrives within 100 ms
test("shows the user's name", async () => {
  loadUser();
  await sleep(100);
  expect(getPageText()).toContain("Ada");
});

// Stable: waits for the actual condition, up to a timeout
test("shows the user's name", async () => {
  loadUser();
  await waitFor(() => expect(getPageText()).toContain("Ada"));
});

Readers ask

What causes flaky tests?

The most common causes are timing problems in asynchronous code, tests that share data or depend on run order, calls to real networks or services, and reliance on random values or the current time. Anything that isn't the same on every run can make a test flaky.

Should you just retry flaky tests?

Retries can keep a pipeline moving in the short term, but they hide the problem and can mask real intermittent bugs in your code. Track flaky tests, quarantine them if needed, and fix the underlying cause.