TLDR
- From trigger to result, Thunders finished the 50 tests in 64 seconds. Selenium took 106 seconds, Cypress 178 seconds and Playwright 196 seconds.
- That makes Thunders 40% faster than Selenium, 64% faster than Cypress and 67% faster than Playwright on the number that developers actually wait for.
- Thunders was also the fastest on execution alone: 64 seconds, against 84 seconds for Selenium, 152 seconds for Cypress and 156 seconds for Playwright.
- The scripted frameworks spent 22 to 40 seconds on setup before and around their tests, and used 2 to 4 billable GitHub Actions minutes per run. Thunders spent no time outside the tests.
- The difference comes from the infrastructure. A standard GitHub runner has 2 vCPUs for the whole run. Thunders runs each test on a distributed browser fleet that is built and tuned only for web tests.
Every developer knows the wait. You push a change, the pipeline starts, and you watch a progress bar. Part of that time goes to the tests. The rest goes to work that tests nothing: a machine starts, the repository downloads, dependencies install, the code compiles, a browser installs.
We wanted a number for that wait. So we took 50 end-to-end tests and ran them four ways: as Playwright, Selenium and Cypress scripts in GitHub Actions, and as Thunders tests in Thunders, an AI testing platform for end-to-end tests. Then we measured two things: how long the tests take to run, and how long the full run takes, from trigger to result.
This article shows the results, explains where the time goes, and explains why Thunders returns the fastest result.
What did we measure?
We wrote the same 50 end-to-end tests four times:
- Playwright, Selenium and Cypress: scripts stored in a GitHub repository, run by GitHub Actions.
- Thunders: tests written in plain language in the Thunders platform, run by Thunders.
For each one, we recorded two numbers:
- Execution time: the time the tests take to run, once everything is ready.
- Full CI run: the time from the trigger to the result. This is the number a developer waits for after a push.
The conditions were the same for all four:
- Application under test: the Thunders web app itself.
- Trigger: a pull request CI for Playwright, Selenium and Cypress. Thunders ran with its built-in scheduler, so it needs no CI job.
- Parallel execution: 10 tests at a time on every framework. On Thunders, 10 parallel runs are available on the Enterprise plan, which has custom parallelization (see pricing).
- Scripted frameworks: a standard GitHub-hosted Linux runner with 2 vCPUs, with the standard settings and caches.
- Numbers: each number is the median of several runs.
What were the results?
| Framework | Execution time | Full CI run (2-core runner) | Time outside the tests |
|---|---|---|---|
| Playwright | 156.1 s | 196 s | 40 s |
| Selenium | 84.2 s | 106 s | 22 s |
| Cypress | 152.0 s | 178 s | 26 s |
| Thunders | 64 s | 64 s** | 0 s |
** Thunders ran with its built-in scheduler, not in a CI job. There is no CI runner to start, so the full run equals the execution time, and the run uses no GitHub Actions minutes.
Thunders has the shortest full run
Trigger to result, in seconds · 50 end-to-end tests, 10 at a time · Scripted frameworks on a 2-core GitHub runner · Lower is better
Thunders
Playwright
Selenium
Cypress"Time outside the tests" is the full CI run minus the execution time. GitHub bills each job in whole minutes, so a run of 106 seconds costs 2 minutes. Thunders runs on its own infrastructure, so its run starts on a prepared browser and uses no GitHub runner minutes.
On the full run:
- Thunders is 40% faster than Selenium (64 s against 106 s)
- Thunders is 64% faster than Cypress (64 s against 178 s)
- Thunders is 67% faster than Playwright (64 s against 196 s), or about 3 times as fast
Where does the time go?
A CI run has two parts: setup and execution.
Execution + time outside the tests
Seconds · Each bar sums to the full run
ThundersPlaywrightSeleniumCypressSetup. Before the first test starts, a scripted pipeline spends 22 to 40 seconds on preparation: start the runner, check out the code, restore the dependencies, install the browsers, build the project. This cost comes back on every pipeline, on every pull request, many times a day. Thunders spends no time outside the tests, because there is no runner to start and nothing to install or build.
Execution. Here the infrastructure makes the difference. On GitHub Actions, all the browsers of a run share one standard runner with 2 vCPUs. Each browser needs CPU to render the page and run its scripts, so the browsers wait for each other, and every test gets slower. Thunders spreads the tests over a distributed browser fleet, so a test does not compete with the other tests for the same 2 cores.
Playwright suffered most from the small runner. We see two likely reasons. Its workers seemed to share the 2 cores less well than Selenium's did. It also captured richer evidence for each test, and evidence capture costs CPU; on a 2-core machine, that cost can slow every other test of the run. Thunders tests are written in plain language and keep working when the interface changes, and they still ran the 50 tests in 64 seconds: 24% faster than Selenium (84 s), and more than twice as fast as Cypress (152 s) and Playwright (156 s).
Put the two parts together and Thunders gives the fastest result from trigger to result, against all three frameworks.
How do you get a faster result?
With 10 tests at a time, execution is most of the run. The direct way to go faster is to run more tests at a time. Here the two models differ.
On GitHub Actions, a standard runner has 2 vCPUs, so more parallel browsers on the same runner compete for the same CPU, and each test gets slower. To go further, you pay for larger runners, or you split the suite into shards on several runners. Each shard is a new machine, so each shard pays the 22 to 40 seconds of setup again, and adds runner minutes to your bill.
On Thunders, the number of parallel runs is a setting of the execution cluster, up to the limit of your plan. The tests run on the Thunders browser fleet, so there is no new machine to prepare and no setup to pay again. With 50 tests at a time, all 50 tests run in one round.
Why is Thunders faster?
1. No build, no browser install, no cold machine
A scripted test is code, and a hosted CI runner is a fresh machine for each job. Before the first test, the runner must start, check out the repository, install or restore the dependencies, build or transpile the project, and install or restore a browser and its driver. A cache makes this shorter, but it does not remove it. A Thunders test is not a code project: it is stored in the platform and ready to run. It runs on infrastructure that is kept ready, with the browsers already in place, so there is nothing to download, compile or install.
2. Proprietary scheduling
With a scripted framework, parallel execution is your job. You set the number of workers, split the suite into shards and size the runners. Thunders schedules the tests itself. Its scheduler starts the next test as soon as a slot is free, so no slot waits for the slowest test of a wave. To run more tests at a time, you change one setting. You do not write a sharding configuration, and you do not pay the setup again for each shard.
3. Logic caching
Thunders tests are written in plain language, and AI turns each step into actions on the page. AI is slow when it has to think about every click on every run. Thunders does not do that. The first run resolves each step, and Thunders caches the result. The next runs replay the cached actions directly, without a call to the model. Thunders calls the AI again only when the page changes and the cached action no longer fits.
4. Smart waits
Scripted tests often wait with fixed delays or long timeouts, because a page that is not ready makes a test fail. A fixed delay is always too long or too short. Thunders waits for the page to be ready, then acts at once. It does not wait a fixed number of seconds when the page is already stable, and it does not act too early when the page is still loading.
5. A distributed infrastructure built only for web tests
A standard GitHub runner is one general-purpose machine with 2 vCPUs. It must build code, run scripts and run every browser of the job on those 2 cores. The Thunders infrastructure is distributed and highly customized, and it has one job: run web tests. The tests spread over a fleet of browser machines, so no test waits for CPU that another test uses. We size the fleet, tune the browsers and operate the whole stack for that job only. This is why Thunders is faster not only on setup, but on execution too.
What does this mean beyond the benchmark?
The benchmark measures one run of 50 tests. In real life, the gap grows in three ways:
- More runs: a team pushes many times a day. The setup time of a scripted framework comes back on every run. With Thunders, it does not.
- Fewer reruns: a flaky test makes a developer run the whole pipeline again. Smart waits and self-healing tests make fewer runs fail for reasons that have nothing to do with the code.
- Less maintenance: a scripted test breaks when a selector changes, and someone must fix the code. A Thunders test heals itself when the interface changes, so the team spends its time on the product, not on the test suite.
A faster result is not only a better number on a chart. It is a developer who stays focused, a pull request that merges sooner, and a release that goes out the same day.
Where Thunders fits in
Thunders is an AI testing platform for end-to-end tests. You describe a test in plain language, Thunders runs it in a real browser, and it keeps the test working when the interface changes. As this benchmark shows, you do not pay for that with speed. You get your result faster than with Playwright, Selenium or Cypress, because Thunders removes the setup that a scripted framework repeats on every run, and runs the tests on a distributed infrastructure built for web tests. When you need a faster result still, you raise the parallel runs with one setting, within the limit of your plan.
You can check this on your own app: take the end-to-end tests that slow down your pipeline, write them in Thunders, and compare the time to result. The free trial runs up to 5 tests at a time; to reproduce the 10 parallel runs of this benchmark, ask for an Enterprise setup in your demo.
Start a free trial or book a demo to see it on your own app.
