Home / Services / Playwright test suite audit
A Playwright test suite audit you can hand to somebody else
An engineer who works in Playwright and nothing else reads your suite and the pipeline that runs it, and writes down what is there.
A Playwright engineer reads your test suite and the pipeline that runs it, and writes up what they find as a report you keep. It is optional, and most clients start straight on the work. It is billed by the hour, from $50 an hour, with no minimum engagement attached.
Most clients arrive knowing the job — a migration, a suite built from scratch, a flaky suite that has to be trusted again — and we scope that and start. The audit is for the team that wants to know what it has before it picks a direction. No engagement here begins with one unless you ask for one.
Asking about one is not booking one. The first email settles whether an audit is worth buying at all.
Who this is for
You have a Playwright suite and you have stopped believing it. It runs on every pull request, it goes red often enough that re-running it is the team's protocol, and the person who wrote it left in the spring. Somebody asks in a meeting whether checkout is covered, and nobody in the room can answer.
- A suite the team inherited and did not choose: three hundred specs, a
helpers/wait.tseverybody imports, and atest.setTimeout(120_000)near the top of the worst file. - A red build the team re-runs on reflex, and a CI bill finance has started asking about.
- A decision waiting on it — rebuild, migrate, or expand the coverage — that nobody can make, because nobody can say what the suite is worth today.
There is usually a second reason for buying one. You may be about to ask for budget, or to defend the suite against somebody who wants it deleted. A report written from outside the team is something you can put in front of a person who does not read specs.
The money could go on the repair instead, scoped from what your team already believes is wrong with the suite — the quicker route, whenever those beliefs hold up. If you already know what is wrong, do not buy this. A team that can name the specs that flake and say why each one does has done the reading, and needs flaky suite repair next. The same goes for a team that has already decided to leave the suite behind: if the question is which framework it lands in, start at migration.
What you get
You get one report, and the headings below are the questions it is written to answer. What any of them says depends on what your suite turns out to be.
Does it cover the flows that earn money?
Which revenue-carrying journeys have an end-to-end test. Which have one that goes green without asserting the thing that matters. Which have none. Named flow by flow, since a coverage percentage does not tell you which journey is exposed.
Which tests flake, and why?
Which tests have failed non-deterministically in whatever run history can be read, and what is behind each one: a race the suite creates, state shared between specs, a locator that matches two elements, a fixed wait standing in for a condition, or an application bug the test is right to catch.
Where does the wall-clock go?
How long the whole run takes, which specs own most of it, and what that time is spent on. Serial execution, no worker parallelism, a login through the UI on every test, a global timeout covering for a slow path.
What does a run cost, and how is it wired?
How the suite sits in the pipeline: shards, workers, retries, which failures block a merge and which are ignored. Any cost figure would be your own CI bill, read back to you.
What should happen first?
What to do, in what order, and what each piece would take. That includes the answer that nothing structural needs doing, and the answer that the next step is work this company does not sell.
You keep the report, and there is nothing in it that only works if you hire us. It is written for two readers: the engineer who has to act on it, and the person who has to fund the acting.
How it works
Access
You give us read access to the repository, whatever CI history you can export or share, and an hour with the person who can explain the decisions behind the suite. That hour decides whether the report can explain anything or can only measure: a hard wait added in a panic before a launch and a hard wait copied off a blog look identical in the diff.
The read
An engineer runs the suite, reads it, and reads the pipeline that runs it — the config, the workers, the shards, the retries, and the runs that went red along with what they went red for.
The write-up
Each finding is written up as the code it is about and the note that goes beside it. Where a finding takes more than a one-line fix, it says what the work is, so you can price it with us or with anybody else.
We publish no duration for this, because there is not one. The read takes as long as the suite and the pipelines take, and the thing that varies most between two suites of the same size is how much of the run history survives: failures that were never retained have to be watched before they can be described.
What a finding looks like
A finding is a piece of your code and a note beside it. Both samples here are written for this page rather than lifted from a client repository, and both are the shape that turns up in a suite nobody has owned for a year.
A fixed sleep standing in for a condition
This test passes. It has never failed, which is where the finding starts:
import { test, expect } from '@playwright/test';
test('shows the dashboard after signing in', async ({ page }) => {
await page.goto('https://example.com/login');
await page.getByLabel('Email').fill('ada@example.com');
await page.getByLabel('Password').fill('hunter2');
await page.getByRole('button', { name: 'Sign in' }).click();
await new Promise(resolve => setTimeout(resolve, 5000));
await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
});login.spec.ts — five seconds of sleep sit between the sign-in click and the dashboard assertion, and they are spent on every run of this spec whether the page was ready in 200 milliseconds or not. That is wall-clock spent doing nothing, multiplied by however many specs carry the same line, and the sleep is also standing in for a check nobody wrote. If the dashboard heading is sometimes slow, the test stays green and says nothing about it, and the day it goes over five seconds the failure arrives with no history behind it. Playwright's web-first assertions retry until the assertion passes or the assertion timeout is reached, five seconds by default, so the assertion on the last line does the waiting on its own.
The fix is the deletion:
import { test, expect } from '@playwright/test';
test('shows the dashboard after signing in', async ({ page }) => {
await page.goto('https://example.com/login');
await page.getByLabel('Email').fill('ada@example.com');
await page.getByLabel('Password').fill('hunter2');
await page.getByRole('button', { name: 'Sign in' }).click();
await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
});State shared between two specs
This is the pair that passes on a laptop and fails in CI:
import { test, expect } from '@playwright/test';
let invoiceId = '';
test('creates an invoice', async ({ page }) => {
await page.goto('https://example.com/invoices/new');
await page.getByLabel('Customer').fill('Acme');
await page.getByRole('button', { name: 'Create' }).click();
await expect(page.getByText('Invoice created')).toBeVisible();
invoiceId = (await page.getByTestId('invoice-id').textContent()) ?? '';
});
test('sends the invoice', async ({ page }) => {
await page.goto(`https://example.com/invoices/${invoiceId}`);
await page.getByRole('button', { name: 'Send' }).click();
await expect(page.getByText('Sent')).toBeVisible();
});invoices.spec.ts — the second test reads an id that the first test assigned
to a module-level variable. Tests in one file run in order in the same worker process, so this
passes today. Turn on fullyParallel, or split the two tests across two files, and the
second one runs in a worker where invoiceId was never assigned: the URL it visits
is /invoices/ and the failure is a missing button. Your team knows this one as the
test that only fails in CI. Playwright's
parallelism documentation is explicit
that tests run in separate worker processes, each with its own isolated browser context, and
that cookies, storage and in-memory globals are already isolated between them. Nothing leaks
from one worker into the other. What breaks is the dependency, because the second
test needs a value the first one wrote into a module in another process. The same shape turns
up whenever a record is seeded once and then mutated by whichever test reaches it first. The
fix is a scope of work: each test creates the data it needs, through a fixture or an API call,
and the change touches every spec that inherits from this one. Playwright 1.62 adds
retryStrategy: 'isolated', which holds failed tests back and retries them at the
end, one at a time in a single worker; it makes a retry easier to read and it leaves the
dependency where it is.
A finding note stops there. How to fix either one yourself is a separate article.
Where this stops
The audit reads the test suite and the pipeline that runs it. "Audit" is a wide enough word that the boundary needs saying.
- It is not a security review. No penetration testing, no dependency CVE list, no opinion on whether the application is safe. Security and penetration testing are not offered by this company in any framework.
- It is not a load or performance test. The report can say how long your suite takes and what a CI run costs you. It says nothing about how the application behaves under load, because Playwright is not k6, JMeter or Gatling and we do not sell load testing.
- It is not a code review of the application under test. We read the tests. If they are hard to write because the application has no stable hooks, that is a finding about testability, and it is not a review of your architecture or a list of defects in your product.
- It is not a review of your QA process. No assessment of test case management, release process or ticket hygiene. That is a different company's offer and we do not take it on.
- An Appium layer is outside what we read. If part of the suite drives a native iOS or Android application, that part has nowhere to land in Playwright. Playwright drives browsers: it emulates a mobile browser, which covers a responsive web app at a phone-sized viewport, and it cannot drive a native application.
What it costs
The audit is priced the way the work is priced, because it is the same engineers doing it. Engineers are billed hourly, from $50 an hour, depending on where the engineer sits. The minimum engagement of one full-time engineer for one month does not apply to the audit: it is bought by the hour, on its own, with no month of anybody's time attached to it.
An hour of it is an hour of one of 75 engineers here, all of whom work in Playwright and nothing else. Which of them reads your suite, and which of the four delivery locations they sit in, is the same question it is on any engagement, and it is answered on the engineers page.
The hours follow the suite: how big it is and how much of it is one shared helper layer every spec imports, how many pipelines and environments run any of it, how much of the CI history can still be read, and whether it runs at all today on a machine that is not the one it was written on. You can place your own suite on that list. Eighty specs in one pipeline with ninety days of retained runs is the short end of it; three hundred across four environments, with failures that only exist in a Slack channel, is the long one.
Read first
- Playwright best practices. The anti-patterns an audit looks for, written as advice you can apply without buying anything.
- Why Playwright tests flake. The self-service answer to why a test fails on one run and passes on the next.
- The trace viewer. Where the evidence behind a finding comes from.
- Parallel runs and sharding. Where the runtime and the CI bill come from.
Questions
Do we have to do the audit before you start work?
No. Most clients arrive knowing the job, and we scope that and start. The audit is for the team that wants to know what it has before it picks a direction, so it is a route in and never a gate. It is optional, it is billed by the hour, and how long it takes depends on the project. The minimum of one full-time engineer for one month does not apply to it.
Why do you charge for the audit?
Because it is engineer hours, and they are billed like every other engineer hour here. What you buy with them is the time of somebody who works in Playwright and nothing else, spent on your suite rather than on a call. The report is yours whatever you decide next: you can act on it with us, act on it with your own team, or take it to another company.
What does a test automation audit cost, and how long does it take?
Engineers are billed hourly, from $50 an hour, depending on where the engineer sits. The minimum engagement of one full-time engineer for one month does not apply to the audit: it is bought by the hour, on its own. We publish no duration because there is not one. That depends on the size of the suite, how many pipelines and environments run it, how much of the CI history can still be read, and whether the suite runs today on a machine that is not the one it was written on.
What do you need from us?
Read access to the repository, whatever CI history you can give us or export for us, and an hour with the person who can explain the decisions behind the suite. The run history is the input that varies most between teams. A suite whose failures live in a Slack channel and somebody's memory can still be read, and less can be said about which of its failures were non-deterministic than for a team with ninety days of retained runs.
What if the report says the suite is fine?
Then it says so, and you have bought an answer to the question that sent you here. A suite that is sound and a suite nobody has checked look identical from a management meeting, and the second one is the reason budgets get spent on rebuilds nobody needed.
Tell us how much of the suite you can see
The number of specs, the CI it runs on, and how far back the run history goes are enough to scope an audit. If you already know what the job is, say that instead and we will start on it: the work does not wait on this page.