While Moore’s law has been slowing for the last 10 years, several developer tools have become popular not necessarily because they can do more, but because they can do the same thing but faster. Vite, Biome & Vitest feel like they are set to replace Webpack, Eslint & Jest. Developers are willing to give something new a chance if it can significantly improve the speed of their feedback loops.

Here at Endform, we want to be the “vite” of playwright. We want to offer you the best way of running end to end tests on the market, and in this post we’d like to explain a little about how we’re doing it.

The challenge with end-to-end testing

Why no Vite for end to end tests yet? End to end tests such as playwright tests aren’t bottlenecked by the speed at which the test code can run, they are bottlenecked by the resources that the browsers they run with consume.

If you’ve seen the effect chrome has on your activity monitor / the amount of RAM your browser regularly consumes, maybe you can visualise the amount of resources it would take to run hundreds of browsers, each testing a different user flow in your application.

With so much compute readily available in the cloud, many companies have found ways of alleviating this problem. Most commonly by sharding over many machines in their CI pipelines. Often, these in-house solutions come with a complexity and maintenance cost, requiring dedicated QA resources or teams to keep things running smoothly.

The ability to run the suite from your local machine flew out the window when you wanted to test more than 10 different user flows.

The current market

There are a few solutions out there that try to tackle this. Most of these are “rent a browser” solutions. The idea is “if the browser is the bottleneck, we can run them for you”.

• You get the server ip/port to a ready booted and running browser on one of their servers.
• You then configure playwright to send commands that control that browser over a websocket.
• Your locally running test code now controls and receives results from the remotely running browser.

For example, if you have a login test, and it says page.goto('myspecialwebsite.com/login'), the framework interprets that as a command to ask the remote browser to open that page. This communication happens over WebSockets, allowing you to both send commands and check the browser's state.

Problems with remote-browser-over-a-websocket

The main problem with this approach is that we have now shifted the bottleneck!

You may notice that many of these providers limit you to running 50/100 remote browsers at a time. Keeping this many websockets alive from a single machine has now instead increased the strain on the network.

Once again, continuing to scale past a certain point, or to run your suite from your wifi-bound laptop, has become hard.

Our new approach

So, how have we at endform tackled this problem differently?

Our favourite property of end-to-end testing code is that it’s usually quite isolated from the rest of your application. The node_modules / js dependencies for just your test code tends to be much smaller than for your entire application.

1. Our rust based cli figures out the exact set of JavaScript file dependencies needed to run each individual test file.
2. We determine the dependency tree for your entire test suite, and calculate which dependencies are common between all tests.
3. Instead of booting remote browsers and sending commands over WebSockets, we boot your test framework on as many individual machines as needed to run all tests simultaneously.
4. We send each machine the necessary dependencies and the specific test to run.
5. We run the Playwright framework from within each machine, loading only one test file per machine.

This approach offloads a lot of work from the machine booting up the test suite. The computing power needed to figure out the dependency sets can easily run on your laptop. Our servers handle the coordination of results, and the heaviest operations - running your framework and a browser - happens within each machine.

The benefits of our approach

So, what does this mean for you?

1. We can run hundreds of end-to-end tests in parallel without hitting the same limits as other browser-based solutions.
2. It's easy to run your suite from your machine without network overhead.
3. We can cache a lot of these operations, as most of your end-to-end tests and their dependencies don't change much over time.

In essence, we believe we've created the world's fastest Playwright end-to-end test runner. It's not just fast - we can scale it reliably too.

What’s next?

We know that running tests fast is only one part of what makes running end to end tests hard. Soon we will be announcing our dashboard that will make it easier for you to maintain your tests over time.

If you’re interested in trying the runner now, don’t hesitate to join our waitlist!