Understanding test results

In this guide, we will use the Endform dashboard to understand the test results.

Real-world tests don’t always pass the first time you write them, or forever after. There is almost no point in writing or running end-to-end tests if we don’t know how to debug them quickly and easily.

We covered in part one of the tutorial how to use the debug flag with Playwright pnpm playwright test --debug. This is a great option for understanding what’s going on in a test while you’re developing it.

In many cases we want to be able to look at test runs after they’ve completed. Let’s take a look at the most powerful options we have available to debug Playwright end-to-end tests after they’ve completed.

Playwright traces

Start by setting the trace option to on in your playwright.config.ts file.

export default defineConfig({
  // Create an HTML report of the test run
  reporter: [["html", { open: "never" }]],
  use: {
    // Always create a trace of the test run even if the test passes.
    trace: "on",
  },
});

Then kick off a run of your suite with pnpm playwright test and boot the HTML report server with pnpm playwright show-report.

Click on one of the tests. Since we had trace set to on, we should have a full trace file for each successful test in our suite.

Here we can navigate the different steps that the test took. We can look at screenshots of what was going on while the test was running. Then we can also inspect parts of the browser while the test was running, including the network tab and console logs from the browser.

Add a flaky test

One of the most common problems you’ll have with your suite is tests that only sometimes fail, otherwise known as flaky tests. Let’s see what it’s like to work with a flaky test in our suite.

tests/flaky-test.spec.ts

import { expect, test } from "@playwright/test";

test.describe("Flaky Test", () => {
  test("should pass most of the time, but sometimes fail", async ({ page }) => {
    await page.goto("/dashboard");
    await expect(
      page.getByRole("heading", { name: "Team Settings" }),
    ).toBeVisible();

    // Let's make this test flaky by randomly failing 30% of the time
    const randomNumber = Math.random();
    expect(randomNumber).toBeLessThan(0.7);
  });
});

Let’s set the trace setting back to retain on failure.

export default defineConfig({
  use: {
    // Keep traces for failed tests
    trace: "retain-on-failure",
  },
});

Now let’s try running the tests again. We will have to run them many times to simulate the flakiness.

pnpm playwright test --repeat-each=10 flaky-test.spec.ts

Take a look at the HTML report. You should have at least one failing test, unless you’re very lucky!

Results in the Endform dashboard

Normally, the problem with flaky tests is that you won’t get the failures neatly collected in a single report. Instead, they’ll be spread out across lots of reports in lots of different runs.

This is where the Endform dashboard can really shine, because it can show you lots of results over weeks and months of historical data.

Let’s run the flaky test with Endform.

pnpm endform test --repeat-each=10 flaky-test.spec.ts

There you’ll get a link to the Endform dashboard where you can also look at the results.

The Endform dashboard is particularly useful for understanding your suite’s performance over time. You can query for things like:

Which test fails most frequently?
Which test is the slowest?
Which test is the most flaky?

You can do this using a combination of the graph view and the results filters. Here for example we have a view of the most frequently failing tests over the last few weeks.

Conclusion

Thanks for following along with our Playwright tutorial!

Our favorite takeaways from this tutorial are:

Spending some time on a good Playwright setup early on saves you a lot of pain further down the road.
Generating tests using the Playwright MCP server is a great way to get going, and it becomes even more powerful if you have the right tools to generate the test data you need.
Don’t forget to take care of your suite as you scale it to more tests. Endform is a great platform to help you do so.