Playwright test maintenance on autopilot with Endform MCP

Do you recognize the button below? Have you ever pressed it? Maybe even felt a little guilty about it?

We all retry failing tests. Everyone does it. Sometimes, we dig deeper. We open the trace, reproduce the failure, figure out what went wrong, and make things better for the next person who hits it.

But we’re all busy. A flaky Playwright test is rarely the most important thing on anyone’s list. It fails, passes on retry, and we move on. A week later, someone else hits the same failure and presses the same button.

This is the kind of work your AI coding agent should be able to take off your plate. Give it the right context and tools, and it can investigate what actually went wrong. Instead of retrying and moving on, it can investigate the flake, fix the underlying problem, and keep it from coming back.

Why end-to-end tests are the worst

Unstable end-to-end test suites are extremely common. It makes sense when you think about how much an end-to-end test depends on. Browsers alone have rendering, focus, timing, animations, and hydration to contend with, across three different engines that all behave slightly differently. Then there is everything around the browser: preview environments cold-start, dependencies slow down, state leaks between tests, and sometimes a third-party API just has a bad minute. When any one of them is slow or behaves slightly differently, a test can fail even when the product is working correctly.

Keeping a suite reliable therefore takes constant maintenance. Most of it is not particularly difficult, just picky and time-consuming. A flaky locator here, a race condition there, an assertion that needs tightening. The problem is what happens when that work piles up. Failures become routine, engineers stop treating red builds as useful signals, and real bugs get easier to miss.

We have written before about what causes flaky Playwright tests and how to fix them. Most failures leave behind the evidence you need to understand what went wrong. The errors, retries, traces, and test history are all there. The problem is finding the time to dig through them and figure out what actually happened.

This is the kind of crap your agent should deal with

Keeping an end-to-end suite healthy used to require an unsung hero: someone willing to spend hours investigating intermittent failures that everyone else was too busy to chase down. Now that job can go to an agent.

But the models powering these agents can only fix what they can understand. An agent needs the history to recognize a pattern and enough evidence to understand what happened. Without that context, you get the classic AI fix:

await page.waitForTimeout(3000);

The test goes green, everyone moves on, and nothing was actually fixed. You are left with slop that hides the real problem and makes your test suite less trustworthy.

History shows the pattern; evidence explains why

A single failure does not tell you much. But when you look across test history, the same failure can start to tell a story. Maybe a test consistently fails on its first attempt and passes on retry, only fails in one browser, or has been getting less reliable over time. One failure is an anecdote. A week of the same failure is a diagnosis.

Once you see the pattern, the next question is why. The agent can inspect the failed attempts and traces to understand what happened, then reproduce the failure on a remote runner when the evidence is not enough. Instead of stopping when the test passes locally, it can inspect the page where the failure actually happened. “Passes on my machine” stops being a dead end.

The Endform MCP server gives the agent three things it needs to follow that path: test history to spot patterns across runs, OpenTelemetry traces to understand individual failures, and remote debugging to inspect the live page when the existing evidence is not enough. That access lets the agent investigate failures instead of just suggesting changes to your test code. It can move from “this locator timed out” to the history, trace, and live page behind the timeout before deciding what needs to change.

Playwright test maintenance with Endform MCP

The idea is simple. MCP (Model Context Protocol) is a standard for connecting AI applications to external tools and data. An MCP server exposes those tools and context to an AI agent through the protocol. With the Endform MCP server, that means giving your coding agent access to the same test history, traces, and debugging tools you would use to investigate a failure yourself. With Endform, that takes a couple of commands.

Authenticate the Endform CLI, then add the local MCP server to Claude Code from your project directory:

npx endform@latest login
claude mcp add --transport stdio --scope local endform -- npx -y endform@latest mcp

It also works with OpenCode, Cursor, VS Code, and other MCP clients that support local stdio servers. For non-interactive environments, set ENDFORM_API_KEY in the server’s environment instead. From there, the agent gets a small set of tools that mirror how you would investigate a failure yourself:

• query_analytics finds patterns in test history, including run counts, pass rates, and durations over time, grouped by test, branch, project, or outcome.

• query_samples pulls the actual runs, attempts, and errors behind those patterns.

• get_otel_trace gives the agent the OpenTelemetry trace for a specific attempt.

• live_test_start, live_test_run, live_test_status, and live_test_stop let it pause a test and inspect the live page.

You do not need to tell the agent which tools to call or in what order. Give it the problem you want investigated, and let it move from the pattern to the evidence.

Three jobs to hand your agent

The best jobs to hand an agent are narrow, repeatable, and easy to review. Do not ask it to fix every flaky test at once. Give it one problem, enough evidence to investigate it, and a clear definition of done.

Fix the worst flake every Friday

Trigger: Once a week, either from a calendar reminder or a scheduled agent run.

Start with a simple routine. Once a week, find the test that needed the most retries, figure out why it is flaky, and propose a fix.

Prompt:

Look at the last 7 days of test runs on main. Which test needed retries most often? Inspect its failed attempts and OpenTelemetry traces, explain why it flakes, and propose a fix. Don’t propose adding waits unless you can show what we would be waiting for. Use the Endform MCP tools.

Tool calls: query_analytics identifies tests with retries over the last seven days, query_samples pulls their recent executions, and get_otel_trace inspects failed attempts when trace IDs are available. If the evidence is not enough, the agent can use live_test_start and live_test_run to test its theory against the live page, then live_test_stop when it is finished.

Answer shape: A specific recurring failure, the evidence behind the diagnosis, and a concrete change that addresses the cause rather than hiding it with an arbitrary wait.

Here is what happened when the prompt was run against an example app with payment, cart, and dashboard flows:

The agent queried Endform analytics for test runs on main that needed retries, using the last seven days of history to begin its investigation.

After reviewing a week of test history, the agent found three tests with the same fail-then-pass pattern and separated them from a fourth test that failed consistently.

The history gave the agent enough evidence to reason about the failures instead of treating each one in isolation. Traces were unavailable for these attempts, so this example relies on repeated failures and their error messages. In the payment test, for example, it identified a likely race around waitForResponse and proposed synchronizing on the actual payment request instead of adding an arbitrary delay.

That is the kind of fix you want: specific, explainable, and grounded in what actually happened. For a deeper look at this workflow, see Investigate test history with AI.

Triage a pull request before pressing re-run

Trigger: Automatically, as a GitHub Actions step when the end-to-end suite fails on a pull request.

When an end-to-end test fails on a pull request, the easiest thing to do is press re-run. But a retry cannot tell you whether the failure is a regression, an expected behavior change, or a flake. You can have a GitHub Action run that investigation automatically, so the agent posts its findings to the pull request before anyone has to decide what to do next.

Add a failure-only step to your existing pull request test job with the Claude Code GitHub Action. This excerpt goes after your checkout, Node.js, and dependency setup. Give the job contents: read, pull-requests: write, and id-token: write permissions:

- name: Run Playwright tests with Endform
  id: e2e
  run: npx -y endform@latest test
  env:
    ENDFORM_API_KEY: ${{ secrets.ENDFORM_API_KEY }}

- name: Triage failed tests
  if: ${{ failure() && steps.e2e.outcome == 'failure' }}
  uses: anthropics/claude-code-action@v1
  with:
    anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
    settings: |
      {"env":{"ENDFORM_API_KEY":"${{ secrets.ENDFORM_API_KEY }}"}}
    claude_args: --allowedTools "mcp__endform__*"
    prompt: |
      Triage PR #${{ github.event.pull_request.number }} in ${{ github.repository }}.
      Use Endform MCP to compare failed tests with 21 days of history on main.
      Comment with evidence: regression, expected change, or flake; retry or investigate.

Passing tests skip the triage step. When the test step fails, Claude investigates and posts its findings; the failed test step still keeps the job red. The condition also avoids launching test triage when an earlier setup step fails.

For both examples, install the Claude GitHub App using its setup guide, and add ANTHROPIC_API_KEY and ENDFORM_API_KEY as repository secrets. Register Endform with claude mcp add --transport stdio --scope project endform -- npx -y endform@latest mcp and commit the resulting .mcp.json so CI has the same MCP server configuration. The action loads .mcp.json automatically; --allowedTools "mcp__endform__*" lets it call Endform tools without an interactive permission prompt. These examples assume PRs from branches in the same repository, where those secrets are available.

Tool calls: query_samples pulls the failed test runs from the PR branch, query_analytics compares the same tests against their pass rates on main over the previous 21 days, and get_otel_trace inspects a representative failure from each group.

The agent will compare the failures on the branch against their history on main and inspect the evidence behind each one.

Answer shape: A comment on the pull request grouping failures into real application failures, expected behavior changes, and intermittent problems, with enough evidence to tell reviewers which are worth re-running and which need human attention.

When the tests fail, the GitHub Action runs the triage and posts the report directly to the PR. Reviewers can see what failed, what the agent found, and whether a retry makes sense, without having to dig through the test results themselves.

You can build this workflow using GitHub Actions and Claude Code, with the same data available through endform analytics, endform samples, and endform live-test.

Suggest test changes on a pull request

Trigger: Automatically, when a pull request changes Playwright test files (*.spec.ts) in the end-to-end test directory.

A test change can fix a flaky assertion, but it can also introduce a new problem or miss something important. Before merging, your agent can run the affected tests in live sessions, explore the application, and check whether the tests still describe how the product is supposed to behave.

Use a separate workflow with a pull request path filter to review changes to your tests. The pattern e2e/**/*.spec.ts includes test files directly inside e2e/ and in its subdirectories:

on:
  pull_request:
    paths: ["e2e/**/*.spec.ts"]

After checkout and your usual install step (npm install, for example), reuse the Claude action above with this prompt and remove its failure-only if condition:

prompt: |
  Use Endform MCP to run the affected tests from PR #${{ github.event.pull_request.number }}
  in ${{ github.repository }} in live sessions. Explore the application and
  check whether the tests need changes. Try proposed changes before recommending
  them, stop the sessions, and comment with findings and anything you could not verify.

Keep the same authentication and MCP settings, and point Endform at your application’s test environment.

Tool calls: live_test_start pauses an affected test, and live_test_run lets the agent explore the application and try changes to the test against the live page. It can use query_samples and query_analytics for historical context, then live_test_stop to end each session.

Answer shape: What the agent explored, any gaps between the tests and the application behavior, and suggested changes it tried in the live session.

If you want to see how the paused-browser loop works in detail, read Debug tests with AI.

Make every failed test worth investigating

A failing test should mean something is worth investigating, not that it is time to press re-run and see what happens. Every time a red build means “probably nothing,” the whole team pays for it. Engineers stop trusting the signal, reviews slow down, releases get held up, and real bugs get easier to miss in the noise.

Keeping the suite stable changes that. You ship faster because green means green, and when something turns red, you have a reason to pay attention. Getting there means treating the data your tests produce as a first-class debugging tool. Every failed attempt, every retry, and every trace is evidence you can use to understand what went wrong and improve the suite.

Agents make that maintenance cheap enough to actually do. Start by asking on demand. Put the weekly flake review on a schedule. Run triage automatically when pull request tests fail, so reviewers have the evidence they need before deciding whether to retry or investigate further.

Run your Playwright suite with Endform, connect your coding agent, and point it at the failures you already have. You actually want a failing test to mean that your application is broken. Now you can afford to make that true.