Flaky Playwright test remediation, so a red build means something again

What this looks like in your repository

None of the code below came out of a client repository. It is written for this page, and it is the shape a flaky suite tends to arrive in. Each sample is followed by what it tells us when we find it in a suite, and by what the register does with it.

A sleep, and an assertion that only looks once

This spec passes most mornings:

import { test, expect } from '@playwright/test';

test('saves the profile', async ({ page }) => {
  await page.goto('https://example.com/settings/profile');
  await page.getByLabel('Display name').fill('Ada');
  await page.getByRole('button', { name: 'Save' }).click();

  await page.waitForTimeout(1000);

  const status = await page.getByRole('status').textContent();
  expect(status).toBe('Saved');
});

Every sample on this page is written against Playwright 1.62.

The version it becomes is shorter, and the sleep is deleted rather than raised:

import { test, expect } from '@playwright/test';

test('saves the profile', async ({ page }) => {
  await page.goto('https://example.com/settings/profile');
  await page.getByLabel('Display name').fill('Ada');
  await page.getByRole('button', { name: 'Save' }).click();

  await expect(page.getByRole('status')).toHaveText('Saved');
});

The one-second sleep is a guess, and the assertion under it reads the DOM once and stops looking. Playwright's documented timeout defaults say what the guess is being made against: an expect assertion retries for five seconds and a test gets thirty, while an action and a navigation have no timeout at all unless one is set. A spec built this way fails on the days CI is busy and passes on the days it is not, and the failure is in the test rather than in the runners, so the row closes inside the test directory. It is also the cheapest kind of row to close, though the ranking still decides when it gets closed: the register is ordered by how often a spec fails, so your build history sets the order the work happens in.

The configuration that decides whether there is any evidence

Before anything can be diagnosed, the failure has to leave something behind:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  retries: process.env.CI ? 2 : 0,
  reporter: [['html'], ['list']],
  use: {
    baseURL: 'https://staging.example.com',
    trace: 'on-first-retry',
  },
});

With trace set that way, a test that fails once and passes on the retry leaves a trace file behind, and npx playwright show-trace opens it. Our reading order is the same every time, and the pane names in it are the trace viewer's own. Actions first, walked to the call that failed; then the Before and After snapshots of the action above it, which is where a locator that resolved to the wrong element stops being a theory; then Network, for a request that arrived after the assertion had given up; then Console and Errors. We come out of it with a sentence naming one of the four causes, and that sentence is the content of a register row.

A suite where trace is 'off', or where CI keeps the last handful of runs, is a suite where the first days go on producing the evidence before any of it can be read. That changes the schedule and the bill, so your config is the first thing we ask to see.

The retry count, and what it does to the meaning of green

Somewhere in a suite that has stopped being believed there is a line like this one, or the --retries=3 flag doing the same job from the command line:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  retries: 3,
});

A retry does not repeat the failure. The argument here is ours, assembled out of what the runner documents itself doing. When a test fails, the runner discards that worker process and its browser and starts a new one, and the retry runs there with beforeAll executed again. A spec that only passes on the second attempt may be passing because it was handed clean state, which means the thing it depends on is state it does not own. Playwright reports those runs as flaky, its own category for a test that failed first time and passed when retried, and the dot reporter prints a ± beside each one. Three retries are three attempts after the first one, so a suite configured like that goes green while the green quietly comes to mean "passed on one of four runs". Nobody agrees to that in a meeting; it accumulates.

Playwright 1.62 added a retryStrategy option, which controls when a failed test is retried. The test configuration reference gives its default as 'immediate' and its other value as 'isolated'. That an option exists for when a retry happens is the framework saying a retry is not a neutral repetition of the test.

Quarantine, written so the count stays visible

A spec that cannot be fixed this month leaves the default run with a tag on it:

import { test, expect } from '@playwright/test';

test('exports the ledger as CSV @quarantine', async ({ page }) => {
  await page.goto('https://example.com/ledger');
  await page.getByRole('button', { name: 'Export CSV' }).click();

  await expect(page.getByText('Export ready')).toBeVisible();
});

The pull request excludes it with npx playwright test --grep-invert @quarantine, and the annotations documentation has a tag as something you filter the test report by, so the number of quarantined specs is a number your team can argue about. test.fixme does the same job where a whole file is going out at once. Both of them are mechanisms, and neither is the deliverable: the deliverable is the line in the quarantine list that says who owns this spec and what has to be true for the tag to come off again.

What it costs

The work here is one engineer reading traces, so the bill is engineer hours and nothing else. Engineers are billed hourly, from $50 an hour, depending on where the engineer sits. The minimum engagement is one full-time engineer for one month.

On a remediation the hours depend on the spec count, how many of those specs have ever failed non-deterministically, how many environments and pipelines the suite runs against, whether traces are already being captured on retry, and how much CI history your runners still hold. A team whose runners keep months of results has a ranked register early, because the failures have already happened and been recorded. A team whose CI keeps the last twenty builds is paying for the same suite to be watched before anything about it can be ranked.