Home / Blog / Playwright visual regression

Playwright visual regression testing: baselines, tolerances and what to snapshot

Updated 1 September 2026 · Playwright 1.62

Three weeks in, a build goes red on a component nobody edited. Someone runs npx playwright test --update-snapshots, the build goes green, and by the third time nobody opens the image at all. The suite still runs on every merge, and it has stopped being able to fail in a way anyone acts on.

Why does a visual test fail when nothing changed?

Two renderings are being compared, and a rendering depends on a great deal more than the product. The visual comparisons guide lists the inputs in a warning box: "Browser rendering can vary based on the host OS, version, settings, hardware, power source (battery vs. power adapter), headless mode, and other factors."

The other half of it is the page rather than the machine. Nothing in the product changed, but a relative timestamp advanced, a list came back from the database in a different order, or an avatar arrived from a service that does not return the same bytes twice.

Some of that the comparator already absorbs, so it is not yours to fix. The assertion "will wait until two consecutive page screenshots yield the same result, and then compare the last screenshot with the expectation", so a page still painting is not a failure. Its animations option defaults to "disabled", which stops CSS animations, CSS transitions and Web Animations before the capture, so a carousel caught mid-slide is not one either.

A suite that is still noisy after all of that is noisy because something inside the frame differs between two runs, or because the run is not happening on the machine the baseline came from.

One case belongs elsewhere. A screenshot captured before the page had settled is an auto-waiting failure in a visual costume, and it is fixed with a web-first assertion in front of the capture — see how Playwright's auto-waiting works.

Where should you generate the baselines?

The filename says where the last one was made. The guide's first-run message is Error: A snapshot doesn't exist at example.spec.ts-snapshots/example-test-1-chromium-darwin.png, writing actual. and it takes that name apart for you: chromium-darwin is "the browser name and the platform", and "Screenshots differ between browsers and platforms due to different rendering, fonts and more, so you will need different snapshots for them".

That example was generated on macOS. The spec further down this page, run for the first time on Windows 11, wrote order-summary-paid-chromium-win32.png instead. Playwright resolves the snapshot filename from the platform it is running on, so a run on a platform that has no baseline of its own is not comparing anything: it reports the snapshot as missing and writes one, which is the message above. If the snapshot folder in your repository is full of -darwin files and your pipeline runs Linux containers, you have just diagnosed your own suite.

The fix is filed on the CI guide rather than the visual one. Under its Via Containers heading for GitHub Actions, running the job inside a container is "useful to not pollute the host environment with dependencies and to have a consistent environment for e.g. screenshots/visual regression testing across different operating systems". So the baselines have to be made inside that same image. Pull it, and run the update in it:

docker pull mcr.microsoft.com/playwright:v1.62.0-noble
docker run -it --rm --ipc=host mcr.microsoft.com/playwright:v1.62.0-noble /bin/bash

# then, inside the container, in your checked-out project:
npm ci
npx playwright test --update-snapshots

Playwright 1.62 · shell

The two docker lines are the documented invocation, copied as published, and they are the one block on this page that was not run here, because the machine the rest of it came off has no Docker on it. The Docker guide gives no volume mount with them, so getting the repository in front of the container is yours to arrange: a bind mount locally, the checkout step on CI. Keep --ipc=host — that page recommends it with Chromium, and says that without it Chromium can run out of memory and crash. Commit whatever comes out of the run, and treat a baseline generated anywhere else as not a baseline.

Pin the tag. The Docker guide is explicit: "It is recommended to always pin your Docker image to a specific version if possible. If the Playwright version in your Docker image does not match the version in your project/tests, Playwright will be unable to locate browser executables." The image is now part of the baseline, so a Playwright upgrade moves the image, and moving the image can move the pixels. Budget a baseline regeneration into the upgrade.

snapshotPathTemplate configures where the files land. Reach for it when the default layout does not suit a monorepo. And if you keep baselines for more than one engine, the browser half of the filename is doing real work — with the caveat that Playwright's WebKit build is not Safari, since it "doesn't work with the branded version of Safari since it relies on patches".

What is worth snapshotting?

Snapshot a component in a state you control. The frame should hold only the things whose appearance is the point, and every extra pixel in it is a pixel that can go red for a reason nobody wants to read about.

Worth a baseline:

• A design-system component in each of the states it ships with.
• The empty state, which is the one nobody looks at by hand.
• The error state, for the same reason.
• Anything that renders money, dates or truncation, where the bug is one character wide.
• A print stylesheet or an email template.
• A layout at a viewport width nobody on the team opens.

Not worth a baseline:

• A dashboard fed by production-shaped data.
• Any frame containing a relative timestamp.
• A list whose order comes out of a real database.
• An avatar, a map tile or an ad served by somebody else.
• A page behind a login whose fixtures reset nightly.
• Anything a text assertion already covers, because toHaveText tells you what broke and a pixel diff only tells you that something did.

Every sample on this page except the Docker invocation above was run against Playwright 1.62 on Windows 11, with the bundled Chromium 151.0.7922.34, over a small local page written for the purpose: an order-summary card that also renders an avatar with a randomised tint, an iframe carrying a live timestamp, and an "updated N seconds ago" line. Every figure quoted below came off those runs, on that machine.

import { test, expect } from '@playwright/test';

test('order summary renders in its paid state', async ({ page }) => {
  await page.goto('/orders/1001');
  const summary = page.getByTestId('order-summary');
  await expect(summary.getByRole('heading', { name: 'Order #1001' })).toBeVisible();
  await expect(summary).toHaveScreenshot('order-summary-paid.png', {
    mask: [summary.getByTestId('last-updated')],
  });
});

Playwright 1.62 · tests/order-summary.spec.ts

The assertion is on a locator, not on the page, so the frame is the card and the chrome around it is excluded. The toBeVisible line in front of it makes the state a fact before anything is captured. The volatile child is named and masked, so the clock inside the card cannot fail the card.

The component-testing guide reaches the same conclusion from the other side and puts it in one line: "Screenshot the returned root locator, not the page, to avoid asserting on anything extra you might put in the gallery." If you are mounting components with mount(), the mounting model itself is covered in the component testing guide for 1.62.

playwright.dev documents a second kind of snapshot. toMatchAriaSnapshot asserts the accessibility tree of a page against a stored template, and fonts, anti-aliasing and the host OS have no bearing on it, which makes it portable across the machines that break pixel baselines. It will not see a colour, a spacing change or an overlap, so the two answer different questions and a suite can carry both.

How do you set a tolerance you can defend?

The page assertions reference defines every option that can loosen a comparison:

The first governs how different one pixel is allowed to be, and it arrives with a value. The other two govern how many pixels may differ at all, and they do nothing until you set them. The conflation to watch for is between threshold and maxDiffPixelRatio, because both take a number between zero and one and they mean unrelated things. Read maxDiffPixelRatio's own definition arithmetically, as a ratio of differing pixels to total pixels, and 0.2 permits one pixel in five to change, which to anyone looking at the diff is the whole image.

Policy of this kind belongs in the config, where it applies to every snapshot in the project and can be reviewed in one place, for the same reason as every other suite-wide setting in the practices worth arguing about.

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  use: {
    baseURL: 'http://127.0.0.1:4318',
  },
  expect: {
    toHaveScreenshot: {
      maxDiffPixels: 100,
      stylePath: './screenshot.css',
    },
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
  ],
});

Playwright 1.62 · playwright.config.ts

Treat that 100 as a starting point; on this bench it was already too loose. Changing one digit of the card's order total, so that the eight in 148.00 became a nine, moved 22 pixels and the run passed. With maxDiffPixels taken back out of the config the same change failed, and the run reported 22 pixels (ratio 0.01 of all image pixels) are different. A tolerance sized to swallow your noise will swallow a change you wanted to see, and on most pages there is no number that separates the two.

So leave threshold where the reference sets it unless one specific snapshot argues for a change, and read a per-test tolerance as a note that the snapshot is unstable.

Masking, and the two ways to do it

For one named element, hand its locator to mask. The reference: "Specify locators that should be masked when the screenshot is taken. Masked elements will be overlaid with a pink box #FF00FF", changeable through maskColor, whose "Default color is pink #FF00FF." In the baseline the spec above produced, 5,880 pixels are exactly #FF00FF: the box over the timestamp, visible to anyone who opens the file.

For a class of element across the whole suite, use stylePath. The guide describes the point of it — "You can apply a custom stylesheet to your page while taking screenshot. This allows filtering out dynamic or volatile elements, hence improving the screenshot determinism." Set it once in the config, as above, and write the rules once:

.avatar,
iframe {
  visibility: hidden;
}

Playwright 1.62 · screenshot.css

With that stylesheet and the one mask entry in place, the spec passed ten times out of ten while the page kept randomising the avatar, the iframe and the timestamp on every load. With the mask and the stylesheet removed and nothing else altered, the same assertion failed ten times out of ten, at between 121 and 1,207 pixels differing per run against a product that never changed.

Masking is cheaper than raising a tolerance, and it is legible in the diff: a reviewer sees the box and knows why it is there, where a number in a config file explains nothing to whoever opens a failure eight months from now. What it costs is coverage. A masked region is a region no test looks at, so mask the clock and not the panel the clock sits in.

Who looks at the failure?

Everything else in a Playwright suite fails with a string a machine can route. This one fails with an image. When the card's status badge gained a single pixel of padding, the run reported Expected an image 462px by 407px, received 462px by 409px. 1821 pixels (ratio 0.01 of all image pixels) are different. It also wrote three PNGs into that test's output directory, order-summary-paid-expected.png, order-summary-paid-actual.png and order-summary-paid-diff.png, printing all three paths in the terminal under Expected:, Received: and Diff:. None of it says whether the extra pixel was wanted.

Somebody has to look, and the trace viewer's Attachments tab is the documented place to do it: "If you're doing visual regression testing, you'll be able to compare screenshots by examining the image diff, the actual image and the expected image. When you click on the expected image you can use the slider to slide one image over the other so you can easily see the differences in your screenshots." Getting the trace off the runner in the first place is covered in the guide to reading a failing run.

A review ends in one of three verdicts. The change was intended, and the baseline is updated. The change was a bug, and the test has just paid for itself. Or the snapshot is unstable, and it should be masked, narrowed or deleted — the verdict teams almost never reach, which is how a folder of baselines nobody trusts accumulates.

The update flag is where that third verdict gets skipped. The CLI reference documents -u or --update-snapshots [mode]: "Possible values are "all", "changed", "missing", and "none". Running tests without the flag defaults to "missing"; running tests with the flag but without a value defaults to "changed"." An ordinary run therefore writes only the baselines that were absent, and a bare -u rewrites the ones that differ. Passing all rewrites every baseline in the project, including the ones that were about to catch something, and it turns the review into a formality. There is also --ignore-snapshots, "Ignore screenshot and snapshot expectations.", which is the right escape hatch for a branch that is mid-redesign and the wrong one for an ordinary Tuesday.

All of it needs an owner, and on most teams the visual specs were added by whoever was annoyed that week. Where nobody holds that job, a suite someone else builds and hands over with a runbook is one way to make it somebody's: the runbook is the deliverable that says what to do when one of these goes red.

What visual regression testing will not catch

It is not design review. A screenshot assertion compares this build against the last one, so a layout that has been misaligned or unreadable since the day it shipped sits inside the baseline, and the suite will defend it for as long as you keep it.

And it is a feature of the JavaScript runner. The API reference says so beside the assertion: "Note that screenshot assertions only work with Playwright test runner." The assertion lists published for Python, Java and .NET do not carry it: no to_have_screenshot, no hasScreenshot, no ToHaveScreenshotAsync. What all three of those lists do carry is the ARIA-snapshot assertion, under each language's own naming. A team running Playwright in Python or Java that wants pixel comparison is looking outside that assertion list for it.

It is also no substitute for a text assertion. The 22-pixel case above went green under a tolerance that a toHaveText on the total would have caught outright.

Questions