Quarantine the flaky Playwright tests you can't fix yet

Your checkout test fails about one run in six. You hit rerun and it goes green. A few weeks in, nobody on the team reads that failure anymore. Everyone just reruns it.

The usual next step is to make the rerun automatic with retries, or to silence the test with test.skip. Retries let it pass on a second or third attempt, so the build stays green. A skipped test stops producing results. You can’t tell whether the flakiness is gone, and it often stays skipped for good.

Quarantine avoids both problems, as long as every quarantined test has an owner and an expiry date.

What quarantining a test actually means

Quarantining a flaky test means removing it from your pass/fail gate without deleting it. The test continues running and reporting results, but its failures no longer block a merge. Those results show whether the test is becoming stable enough to return to the main suite.

Quarantine is defined differently across testing workflows. GitLab’s testing guide, for example, defines it as marking a test to be skipped in CI while allowing it to run locally by default. A test excluded from CI produces no execution results there, limiting the data you have for tracking its stability.

Retry runs a failed test again until it passes or reaches the configured limit. Retries are off by default in Playwright, although the configuration generated by npm init playwright enables two retries in CI. A test that fails first and passes on retry is labeled flaky, and the run still passes unless failOnFlakyTests is set.

A retry gets a fresh worker and browser, but it runs on the same machine as the other workers. If the first attempt failed because that machine was overloaded, the retry can hit the same problem. Passing on the second attempt doesn’t mean the test is fixed. Retries are a stopgap for flaky tests, and a test that passes on retry still needs someone to look at it.

Skip or delete removes the test from execution. Playwright’s test.skip and test.fixme stop it from running, while deleting it removes the check from the suite. You lose execution data and coverage for the part of the application that test checks.

Quarantine keeps the test running but takes it out of the merge gate. While it’s quarantined, a real regression in the flow it covers won’t block your pull request either. It also still uses runner time.

Why “just skip it” and “just add retries” fall short

Skipping stops the test from checking that part of your application in CI. To learn whether a fix worked, you have to restore the test in another pull request and wait for CI, and if it still fails, that pull request goes red for the same issue. A quarantined test avoids that detour because it never leaves the suite.

Retries are useful for occasional failures, such as a network blip. As flakiness increases, retries give unstable tests more chances to pass the build. A guide to Playwright best practices at scale suggests that retries begin hiding the problem when one to two percent of a suite is flaky, though the figure is a heuristic. An engineering post on eliminating flaky tests describes a project whose hour-long suite required at least four or five runs per merge request because of flaky tests.

With two retries, a test gets three attempts and fails the build only if all three fail. At a five percent failure rate, that happens about once in 8,000 runs. At 30 percent, it happens about 2.7 percent of the time, assuming independent attempts. Attempts are not always independent when retries run on the same machine alongside other workers, so resource contention can affect the result.

Skipping or retrying a test also isn’t recorded anywhere as a decision, so nobody is responsible for following it up.

How to quarantine a flaky test in native Playwright

Playwright has no built-in quarantine feature. You can use tags to separate quarantined tests from your blocking suite. Tag the test, exclude it from the blocking job, and run the tagged tests in a separate job.

import { test } from "@playwright/test";

test("checkout flow", { tag: "@quarantine" }, async ({ page }) => {
  // ...
});

npx playwright test --grep-invert @quarantine   # blocking job
npx playwright test --grep @quarantine          # quarantine job

Run the quarantine command in a separate GitHub Actions job with continue-on-error: true. That setting prevents a failing quarantine job from failing the workflow run, while your branch protection requires only the blocking job. At the step level, continue-on-error: true lets the job continue while leaving the failure visible in that step’s output.

name: tests
on: pull_request

jobs:
  blocking:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --grep-invert @quarantine

  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --grep @quarantine --pass-with-no-tests --reporter=html
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with:
          name: quarantine-report
          path: playwright-report/
          if-no-files-found: ignore

The --pass-with-no-tests flag lets the quarantine job succeed when no test carries the tag. Playwright otherwise exits with a “No tests found” error when the filter matches nothing. The job also uploads its Playwright report so reviewers can inspect quarantined failures.

Setup projects need extra care. Playwright’s project docs explain that filters select the primary tests, while project dependencies still run. A tagged test inside a setup project can therefore run during your blocking job. If it fails, the job fails and dependent tests do not run. Keep setup tests untagged, since --no-deps skips every dependency, including setup required by other tests.

Tags keep quarantine state in your code. Adding or removing a test usually means editing the test file, committing the change, getting it reviewed, and running your pipeline. They also do not record ownership, expiry dates, or whether the test has recovered, so those details need separate tracking as your quarantine list grows.

Flaky test quarantine at scale, and where the hand-rolled version breaks

Larger suites fill a quarantine list faster. If each test independently fails one run in a hundred, a 200-test suite passes cleanly only about 13 percent of the time without retries, so more tests end up quarantined just to keep builds green. Keeping track of them by tag gets harder as that list spreads across branches and projects.

You can agree on a review deadline for each entry, such as two weeks, but nothing in a tag-based setup enforces it. If nobody remembers to check, the test just stays quarantined.

Code-based quarantine can also lag across branches. Entries usually live in test annotations or a manifest file, so changing one often requires a commit. If your pipeline builds the branch rather than a merge with the main branch, open branches need a rebase or merge to receive the updated quarantine state. Even on GitHub Actions’ default pull_request trigger, which tests a merge with the base branch, a re-run reuses the original merge commit and can miss the change. Until the branch picks it up, the flaky test can continue blocking it.

If quarantine state lives outside the repository instead, a new rule applies to new runs straight away, and open branches pick it up without a rebase. Endform keeps quarantine rules this way, separate from your test code.

Quarantining tests with Endform

Quarantining according to dashboard data

After you run your suite in Endform, the dashboard shows your tests and their results across runs. Each suite’s page has Top flaky tests and Most failing tests cards, and analytics can filter for tests that pass only after more than one attempt. In the demo project behind this guide, a deliberately flaky checkout test appears in the Most failing tests card, which makes it the first candidate for a rule.

Each quarantine rule belongs to one suite. Within that suite, a rule can match on test ID, Playwright project, branch, or a combination of these fields. An empty field matches every value.

You can create rules from the dashboard’s Quarantine tab or the command line using a suite ID and at least one selector. Creating a rule requires no repository change or rebase. Reason, labeled Context in the dashboard, and expiry are both optional, but setting an expiry when you create the rule gives it a review deadline. The first command below finds the suite and test IDs needed to create it.

npx endform samples test-runs --where 'testRun.outcome = fail' --limit 10

npx endform quarantine create \
    --suite-id $SUITE_ID \
    --test-id $TEST_ID \
    --reason 'Investigating checkout failures' \
    --ends-at 2026-11-20T12:00:00Z

A rule applies to matching tests in every new run from that point on, leaving runs already in progress and past results unchanged. Quarantined tests continue running and consuming runner time, and their outcomes remain in your history. A quarantined test runs once, without retries, even when its project has retries configured, so a test you already know is flaky doesn’t spend your retry budget. A failed quarantined test carries Failed and Quarantined badges, does not count toward maxFailures, and does not fail the suite on its own.

A passing suite can include failed quarantined tests, so judge them by their outcomes across runs rather than the suite status. Infrastructure errors still fail the suite. A quarantined test that fails in a setup project still stops its dependent projects from running, and the suite fails.

A rule that matches on test ID follows that exact test, so renaming or moving the test, or changing its describe blocks, creates a new ID that the rule no longer matches. Rules that match only on project or branch are unaffected. When several active rules match one test, the oldest applies.

The Quarantine tab lists your active rules, with expired and deactivated ones under Rule history. Each active rule’s actions menu lets you edit or deactivate it. From the command line, npx endform quarantine list shows active rules, npx endform quarantine list --all includes inactive ones, npx endform quarantine update replaces an active rule’s settings, and npx endform quarantine deactivate $RULE_ID deactivates a rule.

Managing quarantine automatically with an AI agent

Endform’s MCP server gives your coding agent tools to query analytics and recent test runs, then list, create, update, and deactivate quarantine rules. The same operations are available as commands, so you can run them from a script or CI job.

Keep the quarantine decision under your control and assign each test an owner through a ticket. Define the lookback window and failure threshold in advance, such as failures over the previous seven days. The agent can query recent runs and propose a rule with supporting results and an expiry date. After you approve it, the agent creates the rule and can open a ticket for the responsible team if it has access to your issue tracker.

Linear and Atlassian provide MCP servers that can create tickets, as Linear issues or Jira work items, so the same agent that manages the rule can also reach your tracker. Add the ticket ID to the rule’s reason so you can trace the rule back to its owner. The expiry date sets the review deadline. Endform provides the test queries and quarantine-rule operations, while your agent client and issue tracker handle approval and ownership.

When to remove a test from quarantine

A quarantine is over when the test is fixed and back in the blocking run. If it no longer checks anything useful, rethink what it should test.

One passing run is weak evidence of stability. A test that fails 10 percent of the time has a 90 percent chance of passing once, while its chance of passing twenty consecutive runs is about 12 percent, assuming independent runs.

Decide how many consecutive passing runs a test must complete before it returns to your blocking run. Setting that threshold when the test enters quarantine gives you a consistent return condition. You should also give every quarantine entry an expiry date, which sets a deadline for reviewing the results.

Endform’s Quarantine tab applies a default threshold for you. Each active rule gets a review status: Reconsider once every test and project it covers has at least 20 passing runs over the last seven days with no more than 5 percent failing or needing retries, Still needed while failures stay above that, and Needs more evidence until there is enough history to judge. Rules that are ready to come out are listed first.

When a rule’s expiry (--ends-at in the CLI) passes, it stops applying to new runs. The test’s failures then count toward your blocking run unless another rule matches. If the test needs more time, edit the rule to push its expiry back before it passes. An expired rule can’t be edited, so after that point you create a new one.

A rule without an expiry remains active until you deactivate it, leaving no scheduled review point. You can ask a coding agent to review recent test results before deactivating a quarantine rule. In one example, you ask an agent whether a checkout test has recovered over the previous seven days. It finds 18 consecutive passing runs, including six after the underlying issue was addressed, and deactivates the rule.

Red should still mean broken

Quarantined, the checkout test from the opening stops blocking merges but keeps running, so you can see when a fix makes it pass. When the rule expires, its failures count again unless someone extends it. Meanwhile every other red build still means something broke.

To try this in Endform, run your Playwright suite with npx endform test, then create a quarantine rule with an expiry date from the Quarantine tab or with the quarantine create command.