Home / Blog / Playwright in GitHub Actions

Playwright in GitHub Actions: the decisions the docs leave you

Updated 1 September 2026 · Playwright 1.62

The workflow file on Playwright's CI page is twenty-seven lines long and goes green on the first push. Its companion, the walkthrough for setting CI up, takes an empty repository all the way to a report sitting in the artifacts tab. Both are free and both are correct, and they are what a search returns first. Copy one and get on with the release.

Week two is where it starts costing. The job that took four minutes the day you added it now runs long enough that people open another tab, the report is a zip nobody has downloaded since, and one spec goes red on Friday afternoons with the re-run button as the entire response. The documentation gave you a working file and stopped there, because a file is not a set of decisions.

The samples here target Playwright 1.62 and were exercised on the 1.62.1 patch from npm, under Node 20.19.6 and TypeScript 5.9.3 on Windows 11. The workflow files were never executed as GitHub Actions runs, because this machine has no runner; the note under each one says what was done to it instead.

What should run on every pull request, and what should run nightly?

Hardly anybody makes this split on purpose. One workflow file grows until it runs everything on every push, and then somebody asks in standup whether the tests could move to nightly, which is the sound of a suite about to be switched off.

The split that survives: a pull request runs one browser project and the tests the change could plausibly have broken, while the whole suite, every project and every spec, runs when the branch lands on main and again overnight, where an hour bothers nobody.

--only-changed is the mechanism for the first half, and the documentation is candid about it. docs/ci says the flag analyses the suite's dependency graph, calls that a heuristic, warns that it might miss tests, and tells you to run the full suite afterwards regardless. Carry the caveat: this buys a fast pull request, and the price is a missed test surfacing at merge.

The flag compares against a git ref, so the checkout has to have one. Clone with --depth=1 and it refuses, naming the cause:

Error: The repository is a shallow clone and does not have 'origin/main' available locally.
Note that GitHub Actions checkout is shallow by default: https://github.com/actions/checkout

Playwright 1.62.1 · reproduced locally against a shallow clone with no base branch · the process exits 1

Hence fetch-depth: 0 on the checkout step.

name: Pull request tests

on:
  pull_request:
    branches: [main]

concurrency:
  group: pr-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  changed:
    name: Tests this pull request touched
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7.0.1
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v7.0.0
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps --only-shell chromium
      - run: npx playwright test --only-changed=origin/$GITHUB_BASE_REF --project=chromium
      - uses: actions/upload-artifact@v7.0.1
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 7

GitHub Actions · Playwright 1.62 · YAML validated against the workflow schema, every action input checked against that action's own action.yml at the pinned tag, Playwright commands run locally · not executed as an Actions run

cancel-in-progress: true kills the run for the previous push the moment a new one arrives, so a branch that gets four commits in ten minutes runs the tests once. timeout-minutes: 15 is the next section's argument written down.

Pin the action versions and re-check them. Playwright's CI page currently prints checkout@v6, setup-node@v6 and upload-artifact@v5; the walkthrough beside it prints upload-artifact@v4; GitHub's own artifact tutorial prints @v4 too. Asked directly on 1 September 2026, the three repositories answered v7.0.1, v7.0.0 and v7.0.1 as their latest releases. A floating @v4 looks exactly like this from the outside: it still works, and it has quietly aged.

How long is a pull request allowed to take?

A pull-request check gets about as long as the person who opened it will sit and watch. Past that they switch context, and a check nobody is watching gets read as paperwork: re-run without being read, then moved to a nightly cron, then deleted.

So choose the window and make the pipeline fit inside it. timeout-minutes is where that choice becomes enforceable, and the value the documentation ships is 60. A pull-request job permitted to run for an hour will find a way to take an hour.

Most of why the same suite is slower here than at home is one line of configuration. docs/ci asks for workers set to 1 on CI, to prioritise stability and reproducibility, so that each test has the full system resources; the same paragraph closes by sending you to sharding for wider parallelism. Stop reading at the first and you have a suite running one test at a time, with nothing in the log to say so. The recommendation is sound and it is priced: you pay for every test end to end.

From GitHub's limits reference, so that nobody plans past them: a job on a GitHub-hosted runner is terminated at six hours, a job matrix generates at most 256 jobs per workflow run, and concurrent jobs on standard GitHub-hosted runners are capped by plan at 20 on Free, 40 on Pro, 60 on Team and 500 on Enterprise. That last figure ends the "just use forty shards" idea.

How many shards should you actually pay for?

If the repository is public, none of them — the minutes are free. GitHub's billing documentation is explicit: "GitHub Actions usage is free for standard GitHub-hosted runners in public repositories, and for self-hosted runners." Everything below about cost therefore assumes a private repository, where each account gets a quota of free minutes by plan and anything past it is billed at standard rates. On a public repository the only ceiling left is the concurrency cap, so the question stops being how many shards you can afford and becomes how many will actually run at once.

Shards buy wall clock and they cost minutes. Four of them means four checkouts, four npm ci runs and four browser installs for one suite's worth of tests, plus a job at the end to merge the reports. The tests divide between the jobs; everything that is not tests happens once per job.

The number falls out of two timings you have to take yourself: how long the tests need when one job runs all of them, and how long that job spends on everything else. The measured worker curve and the shard arithmetic works both out and shows where the next shard stops earning anything. GitHub puts a ceiling over that arithmetic: shards beyond your plan's concurrent-job cap queue. Past the cap you are paying for more jobs and the person waiting on the pull request sees no improvement.

Here is the other half of the split, with the matrix in it.

name: Full suite

on:
  schedule:
    - cron: '17 3 * * *'
  push:
    branches: [main]

jobs:
  full:
    name: Shard ${{ matrix.shard }} of 4
    runs-on: ubuntu-latest
    timeout-minutes: 60
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v7.0.1
      - uses: actions/setup-node@v7.0.0
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blob
      - uses: actions/upload-artifact@v7.0.1
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report/
          retention-days: 3

GitHub Actions · Playwright 1.62 · schema-validated, action inputs checked at the pinned tags · not executed as an Actions run

fail-fast: false is there because the matrix default is true, and on true GitHub cancels every in-progress and queued job in the matrix as soon as one of them fails — leaving a red build and three quarters of the evidence missing. The cron avoids the top of the hour because GitHub says scheduled events can be delayed when load is high, and the start of every hour is when load is high.

The file stops one job short on purpose. Four shards write four blob reports and no single answer about the build, and turning them into one report is npx playwright merge-reports --reporter html ./all-blob-reports in a job of its own, with a fixed cost of its own.

Where do the browsers come from?

Three ways in, and the documentation argues against the one most teams reach for first.

Install per run. npx playwright install --with-deps fetches the browsers and their Linux system dependencies on every run. Two flags cut it down and both are free: name a single browser, and add --only-shell when nothing sets a channel, which skips the full Chromium download in favour of the headless shell. That is why the pull-request job above installs --with-deps --only-shell chromium while the nightly one installs everything.

The container. mcr.microsoft.com/playwright:v1.62.0-noble as the job's image, with the documentation's options: --user 1001. The browsers and the dependencies are already inside, so you pay an image pull rather than an install step, and the tag pins the browser build to a Playwright version.

container:
  image: mcr.microsoft.com/playwright:v1.62.0-noble
  options: --user 1001

GitHub Actions · the two keys go on the job, and the install step comes out · image and options transcribed from docs/ci, not run here · tag confirmed present in the registry · the merged job validates against the workflow schema

Check that the tag exists before you print it. The registry lists both v1.62.0-noble and v1.62.1-noble, and the documentation prints whichever version the docs site is currently serving, which need not be the one in your package-lock.json. What ships inside that image decides whether the pull is worth it.

Cache the browsers. docs/ci argues against it, in its own words: "Caching browser binaries is not recommended, since the amount of time it takes to restore the cache is comparable to the time it takes to download the binaries. Especially under Linux, operating system dependencies need to be installed, which are not cacheable." If you do it anyway, the same section says to key the cache against a hash of the Playwright version.

Cache the part that does cache well. The cache: npm input on actions/setup-node handles the package manager's own directory, keyed on your lockfile. Browsers are no part of it.

A WebKit project in CI exercises the WebKit engine and stops there. Playwright's browsers page states that it does not work with the branded version of Safari, since it relies on patches, so a green WebKit shard says nothing about Safari on anybody's phone.

How does the report get out of the runner and into somebody's hands?

The artifact nobody opens is a distribution problem. The documentation's workflow gives you playwright-report/ uploaded by actions/upload-artifact: a zip, behind a download, on a page most of the team has never scrolled to. Count the steps between a red job and a person seeing the cause: open the run, scroll to the artifacts, download the zip, extract it, serve it. Five, and nobody takes them.

The last two collapse into one. npx playwright show-report accepts a .zip directly, provided the archive has index.html at its top level, and it extracts and serves it for you — checked here by zipping a report and serving it at localhost:9323. Opening index.html off the disk still fails, because the report is an application that reads data files sitting beside it.

retention-days: 7 in the pull-request job is the clock on all of it. The value cannot exceed the retention limit set by the repository, organisation or enterprise, and if you leave it out GitHub keeps build logs and artifacts for 90 days. Artifact storage is capped by plan as well, at 500 MB on Free and 50 GB on Enterprise Cloud. Three tests across three browser projects, with three traces in them, made a 1.8 MB report on this machine; real suites are not three tests.

Sharding changes the shape of this. Until the blob reports are merged there is no single report to upload, which is why the nightly file above uploads four artifacts and no HTML at all. The reporter list says which one to point at which reader.

The route out of the artifacts tab is to publish the report somewhere with a URL. The documentation gives exactly one, to Azure Storage static website hosting, using a service principal and three repository secrets, and it notes that the step will not work for pull requests from a forked repository, because those runs cannot read your secrets.

The trace is the artifact worth distributing. With trace: 'on-first-retry' a trace exists only for a test that failed and then ran again, which is precisely the run somebody needs to look at. npx playwright show-trace path/to/trace.zip opens one locally, and trace.playwright.dev opens one in a browser — the docs state that the viewer loads the trace entirely in your browser and transmits no data externally. Reading a trace you have never opened before has a reading order, and it is four moves long.

What should a failing job say to the person who broke it?

The default reporter on CI is dot. A wall of dots with a stack trace underneath is a correct machine record and a poor message to a colleague.

The github reporter puts the failure on the pull request itself, as an annotation against the line that threw. It is not on by default. Running the config below with CI=1 emitted this, one line per failure:

::error file=tests\a.spec.ts,title=[chromium] › tests\a.spec.ts:8:5 › fails first, passes on the retry,line=10,col=43::

Playwright 1.62.1 · the annotation line with the message body cut; the run emitted one per project, each carrying its project name inside title

docs/test-reporters advises against that reporter under a matrix strategy, because the stack traces multiply across the shards and bury the file view. So it belongs on the pull-request job and not on the sharded nightly, which is one more decision the split in the first section pays for.

Re-running until green is a decision too, an undeclared one taken by whoever clicked the button and recorded nowhere. Retries in the configuration are the declared version: the run reports what needed a second attempt, and the count stops being invisible.

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  forbidOnly: !!process.env.CI,
  failOnFlakyTests: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  retryStrategy: 'isolated',
  reporter: process.env.CI
    ? [['github'], ['html', { open: 'never' }]]
    : 'list',
  use: {
    trace: 'on-first-retry',
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
    { name: 'firefox', use: { ...devices['Desktop Firefox'] } },
    { name: 'webkit', use: { ...devices['Desktop Safari'] } },
  ],
});

Playwright 1.62 · TypeScript · run on 1.62.1, all three projects, and type-checked under --strict

retries: 1 means at most two attempts, which is enough to record a trace and not enough to grind a broken build into a green one. retryStrategy: 'isolated' arrived in 1.62 and moves the retries to the end of the run, one after another in a single worker, so a retry is not competing with the rest of the suite for the machine; the default is 'immediate'. failOnFlakyTests stops a pass on the second attempt from being reported as a pass. forbidOnly catches the test.only that would otherwise turn a full run into a run of one test, with a green tick on it.

What that does to a run, on the three-test suite used throughout this page: nine tests, one test failing its first attempt in all three projects and passing on the retry, a summary reading 3 flaky and 6 passed, three traces written and none for the six that passed, and a process exiting 1. Drop failOnFlakyTests and the same run exits 0 — green in the tab, flaky in a report nobody opens. Diagnosing why that test is intermittent is a different job, and it does not start in the workflow file.

Getting all of this decided on a pipeline nobody currently owns is work we take on: a client hands us the workflow files and the suite, and we build the Playwright side of their CI and hand it back with a runbook for whoever is on call.

What about secrets?

Put credentials in repository secrets and reference them as ${{ secrets.NAME }}: you knew that. The half that gets skipped is in docs/ci-intro, and it is about the artifacts rather than the workflow. Trace files, HTML reports and console logs contain whatever the test execution contained, which can be a test user's credentials, an access token for a staging backend, or your own source. Upload them only to artifact stores you trust or encrypt them first, and treat sending a report to a colleague as the same act.

It passes locally and fails in GitHub Actions

None of these is bad luck; each traces back to a decision above.

• One worker there, several here. A different number of workers is a different execution order, and a spec that depends on the order finds out on the runner first.
• Everything is headless. Locally you have probably been watching it.
• The container and ubuntu-latest are not the same machine. Different fonts and locale change how the text your assertion reads is laid out — one more thing a pinned image holds still, and a good part of what the pull is for.
• Timeouts calibrated on a laptop with its CPU to itself. A shared runner with several shards on it is a slower machine, and the margin you never measured is the one that goes.
• The application under test is not up yet when the first test starts, so the first spec in the run fails and the rest pass.

The first move is the same in every case: open the trace from the failing run before you touch the workflow file.

Questions