Speeding up with Endform

In this guide, we will use endform to speed up your test suite.

Note

To start from this step, checkout the stage-2-generated-tests branch of the tutorial repository. For more information about getting started with the tutorial repository, go here.

At this point in time, we have a pretty decent Playwriht end to end testing suite that can test most of the dummy features in this application. We can test complex multi user flows like inviting, and we have the ability to create new users on demand when the user data conflicts with other tests.

Since this a very simple application, our tests are thankfully very fast to run. Let’s explore their performance.

Running the tests

We are familiar with running the tests with Playwright:

pnpm playwright test

Let’s try running the tests with Endform also to get a benchmark:

pnpm endform test

On my machine (M2 Macbook Pro), the tests ran in 20 seconds with Playwright, and 25 seconds with Endform.

With these very fast, very small, and few tests - uploading and running the tests on Endform’s remote runners aren’t giving us much of a speedup.

Cranking up the heat

Let’s see what happens when we run more tests.

pnpm playwright test --repeat-each 2

pnpm endform test --repeat-each 2

Here, Playwright took 30 seconds, and Endform took 25 seconds.

pnpm playwright test --repeat-each 5

pnpm endform test --repeat-each 5

Here, Playwright took 30 seconds, and Endform took 25 seconds.

If we plot the results of different repeat counts, we get:

Once we start to add more tests (or if the tests had taken longer to run), Playwright will keep slowing your suite down, while Endform will keep the speed constant.

Other influential factors

This is one case of the speed of this particular suite, but lots of things have an impact on the speed of a suite. Like:

• How reliable or flaky the tests are
• How long the tests take to run
• How slow the slowest test is
• How performant the app the tests are run against is

We think that we have the fastest Playwright test runner on the market, but we’re really curious to hear what your experience in running Endform versus Playwright in your test suite is like.

Running in CI

Let’s make a CI workflow that runs the tests with Endform.

.github/workflows/e2e-tests.yml

name: Endform tests on main

on:
  repository_dispatch:
    types:
      - 'vercel.deployment.success'

jobs:
  e2e-tests:
    timeout-minutes: 5
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up pnpm
        uses: pnpm/action-setup@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: lts/*
          cache: 'pnpm'

      - name: Install dependencies
        run: pnpm install

      - name: Run Playwright tests with Endform
        run: npx endform@latest test
        env:
          ENDFORM_API_KEY: ${{ secrets.ENDFORM_API_KEY }}

Here we used the repository_dispatch event to trigger the workflow when the Vercel deployment is successful.

Some other thigns you could try are:

• Just trigger on commits to the main branch
• Triggering on pull requests
• Sending a message to a Slack channel when the tests are successful or fail