Build vs Buy Test Automation: What Costs Less to Maintain?

Build vs buy test automation is a long-term operating-model decision. The real comparison goes deeper than setup effort versus licence cost: it includes the maintenance burden year over year, the evidence each model produces when something breaks, how well either fits the existing stack, and who owns reliability when things drift. This article gives you a framework for collecting that evidence. It also shows, criterion by criterion, how one buy option, Thunders, answers each question, so you have a concrete benchmark to hold any platform against.

‍

What building custom QA automation really involves

Building a custom QA automation framework means owning the full stack: framework design, test authoring, environment configuration, test data management, CI/CD integration, reporting, and documentation. Every one of those layers needs ongoing attention. The cost of implementing a framework is a one-time figure. The cost of operating it is not.

When the application under test changes, whether through new UI elements, refactored workflows, or updated APIs, the framework must be updated to match. That work recurs at whatever cadence the application ships.

The maintenance tax behind custom scripts

The maintenance work falls into predictable categories. Locator updates are the most visible: a selector changes and tests fail until someone repairs it. Changed workflows require test logic to be rewritten, not just patched. Flaky tests require investigation to separate genuine defects from environment instability, timing issues, or test-data side effects.

That locator loop is the one AI test agents were built to remove. In Thunders, a test step is stored as the user action it verifies ("add the product to the cart and check the total") rather than as a CSS or XPath selector, so a renamed button or a moved element doesn't fail the run. The agent resolves the intent against the current page.

Beyond the tests themselves, CI/CD integration work accumulates: pipeline scripts, environment provisioning, parallelisation configuration, and reporting integrations all need attention when the infrastructure or deployment process changes. Onboarding a new engineer to the framework takes time, particularly when conventions are undocumented or live in one person's head.

The table below separates the main cost categories, the questions to ask before estimating them, and the evidence worth collecting when comparing this baseline to a platform.

Collect this evidence from your own history. Industry averages vary too much by team size and application type to stand in for your team's actual burden. If you trial a platform, track the same six categories during the trial so the comparison is like for like.

When building is the right choice

Building is the right choice in a specific set of conditions:

• The application uses specialised protocols, unusual hardware interfaces, or deployment environments that available platforms do not support.
• Strict data-residency, security, or regulatory requirements rule out third-party test execution.
• The team has deep, sustained expertise in the framework it maintains, and the operational cost is understood and budgeted.
• A necessary behaviour, such as precise timing control, a proprietary communication protocol, or integration with internal tooling, cannot be replicated on any available platform.

In these scenarios, a custom framework is not a legacy choice. It is the only viable one. The team owns reliability, documentation, upgrades, repair evidence, and the cost of every change, and that trade-off is acceptable when specialised fit cannot be achieved any other way.

Check the second condition before you commit, though. Data residency and security no longer rule out a platform by default. Thunders, for example, can be deployed in a dedicated single-tenant cloud, self-hosted, or fully on-premises with your own AI model keys, and is ISO 27001, SOC 2 and GDPR compliant. Mobile runs on real iOS and Android devices can be kept in the US or the EU. If that covers your constraint, it stops being a reason to build and becomes an evaluation question.

When a hybrid model is the right choice

Some teams do not face a clean binary. Different parts of the test suite carry different risk profiles, change rates, and control requirements.

A hybrid keeps coded tests for paths where specialised control, performance measurement, security validation, or regulatory compliance is required. It moves the broader set of user-facing workflows, the ones that change frequently and generate most of the repair work, to a platform. A typical split: a proprietary-protocol suite stays in code, while customer-facing web flows move to Thunders, which runs as a step in the same CI/CD pipeline (GitHub, GitLab, Jenkins), so both layers gate the same release.

The risks are real. Two maintenance systems can diverge: reporting splits, ownership boundaries blur, and the cost of running two layers can exceed the cost of choosing one. A hybrid makes sense only when the team can define a clear rule for which tests belong where, how evidence is consolidated, and who is accountable for each surface. Without that clarity, a hybrid adds operational complexity without reducing work.

‍

What buying a QA automation platform changes

A QA automation platform shifts the operational model. Instead of building and owning the infrastructure that runs tests, teams configure tests within a vendor-managed environment and rely on the vendor for execution, integrations, and updates. Framework upkeep goes down, but buying does not eliminate maintenance. It can relocate some of it to the vendor and introduces vendor dependency, migration costs if the platform does not fit the workflow, and the risk that the platform drifts from the team's requirements over time.

How much maintenance actually disappears depends on what the platform stores as "a test." Record-and-replay tools still produce selectors under the hood, so they mostly relocate the locator problem. Agent-based platforms such as Thunders store the intent of each step and let an AI agent resolve it against the live UI at run time. That is a different maintenance model, and it is the one to test hardest during a trial.

Buying is worth evaluating seriously when:

• The application has a standard web-based workflow that changes frequently.
• The team lacks capacity to maintain framework infrastructure alongside feature delivery.
• People outside QA engineering (product managers, developers, business analysts) need to write or review tests, which plain-language platforms like Thunders make possible.
• Auditability, supported integrations, and governance tooling are evaluation priorities.
• The team can validate, through a trial on a representative workflow, that the platform reduces repair work rather than relocating it.

Buying is a poor fit when the application has specialised constraints the platform cannot address, when vendor dependency conflicts with security or procurement policy, or when the platform's evidence and audit trail do not meet the governance requirements the team must satisfy.

Vendor lock-in: the risk to settle before you sign

The biggest long-term risk of buying is lock-in. If your tests exist only inside a vendor's platform, leaving means rebuilding the suite from scratch, and that exit cost quietly grows with every test you add. It also hands the vendor leverage at every renewal, and it is often the reason procurement, legal, or a vendor-risk audit blocks a platform outright.

Ask every vendor one direct question before committing: can we take all our tests with us, as code we can run without you? Accept only a demonstrated answer. A CSV of test names or a PDF report is not an exit path.

Thunders answers yes. It exports all your test cases as a standard Playwright TypeScript project that runs with npm ci and npx playwright test, on your own machines and in your own CI, with no dependency on Thunders. Each Thunders step becomes a named Playwright step with your original plain-language instruction kept as a comment above its code. Reused tests become shared modules, each environment gets its own file, and secrets are never copied: they are listed, empty, for you to set. An export report shows which steps exported as full code and which need manual work. Cloud customers request the export from their Thunders contact; self-hosted customers run it themselves.

Be clear about what any export can and cannot carry. Plain code cannot do what Thunders does with AI at run time: plain-language checks, self-healing selectors, generated test data, and accessibility personas export as commented TODOs, and native mobile tests are listed rather than converted, because Playwright does not run mobile apps. That is the honest shape of lock-in risk with an AI platform: you keep your tests, and you take back the maintenance the platform was doing for you.

Self-healing test automation: evaluate the evidence, not the label

Self-healing is the term vendors use to describe automated adaptation to UI or workflow changes. The label covers a wide range of behaviours, from simple selector fallbacks to AI agents that reinterpret a step from its intent.

The label alone is not enough to evaluate. What matters is what happens when a test breaks:

• What exactly changed? (Element, selector, visual structure, workflow path?)
• Why was the change made? Was it intentional or a regression?
• What evidence was captured to support the automated repair?
• Can a reviewer see, approve, reject, or roll back the change?
• Is there a version history and a permissions model governing who can accept repairs?

A system that silently modifies a test and returns a green result without preserving evidence makes it harder to tell whether the original test intent is still being verified.

As a reference point, here is how Thunders handles it. Self-healing works from the intent of the step, not from a single locator. Before a run, Thunders AI flags fragile steps and proposes improvements; after a run, it suggests step updates when the product has evolved. Each suggestion can be applied or dismissed by a reviewer. When Thunders heals a step during a run, the run report shows every healed step and what was changed to keep it passing, and the run summary counts them, so nothing is healed silently. Test case versions can be restored, and teams can edit tests on a separate Branch, review the changes step by step, and merge them into main. Admins scope each user's access by project, with a Read-only role for people who should see tests but not change them. When a run fails, Thunders AI investigates it, pinpoints the failing step, separates real product bugs from test issues, and drafts a bug report ready to file in Jira or Linear.

Hold every vendor, Thunders included, to the same standard. Require a demonstration on a representative workflow, not a curated demo environment, and ask specifically about failure modes: what happens when a repair is wrong, and how is a misrepaired test detected and corrected?

Keeping the existing stack in view

Most teams evaluating a platform do not start from zero. They have existing coded tests, CI/CD pipelines, and reporting tooling that cannot be abandoned overnight.

Evaluate any platform against: import paths for existing test suites, API and CLI access for CI/CD integration, export options if the platform relationship ends, and coexistence with coded tests in a shared pipeline. For teams with an existing coded suite, the Selenium documentation is the primary reference for understanding what migrates and what does not.

To make this concrete: Thunders generates tests from plain-language prompts, URLs, Jira tickets, Linear issues, product documentation, and Figma designs. It runs inside existing CI/CD pipelines with live run streaming, connects to Jira, Xray, Linear, GitHub, GitLab and Jenkins, and exposes an MCP server so tests can be created and run from AI assistants such as Claude, ChatGPT, or Cursor. If the relationship ends, Thunders exports all your test cases as Playwright code (see the vendor lock-in section above).

Whether a platform replaces, supplements, or runs alongside coded tests depends on the workflow and the risk model. That decision belongs to the team, not to the vendor.

‍

Build vs Buy Test Automation: A Practical Decision Framework

The table below is a starting point, not a verdict. Different teams will weight these criteria differently. Fill in your own evidence before reaching a conclusion.

A TCO model should include internal engineering time for framework maintenance and repair, CI/CD and infrastructure costs, platform licence fees, onboarding, support, switching costs, and governance overhead. Run it over three to five years, not one, and use your own team's actual data. Published benchmarks vary too much by team size and application complexity to substitute for real inputs.

Example decision paths

Build: The application uses a proprietary binary protocol no available platform supports. Two senior engineers built and maintain the framework. The operational cost is understood, budgeted, and outweighed by the control and specialised coverage the custom suite provides.

Buy: The team maintains a standard enterprise SaaS product. UI changes ship fortnightly, and engineers spend a material part of each sprint repairing locators and chasing flaky tests. They trial Thunders on two representative flows, checkout and onboarding. The QA lead and a product manager describe the scenarios in plain English, the tests run in the existing CI pipeline, and the team compares repair incidents and rerun rate against the previous quarter before committing.

Hybrid: A regulated financial application has a core transaction flow that must be tested against a specific API version, with evidence auditable for three years. That path stays coded. The surrounding user-facing workflows, which change frequently and generate most of the locator-repair work, move to a platform. Because test data cannot leave the bank's environment, the team shortlists only platforms with self-hosted deployment, such as Thunders. Ownership rules and evidence consolidation are documented before the split is implemented.

Questions to ask in a platform evaluation

• How does the platform handle a changed UI or workflow, and what evidence is retained for every automated change?
• Can reviewers see, approve, reject, and audit automated repairs, and what is the permissions model?
• What integrations, import paths, APIs, export options, and CI/CD triggers are supported, and what are their limitations?
• How are test versions, evidence, permissions, failures, and rollback managed?
• Can we export all our tests as runnable code (for example Playwright) that works without the platform, and what does not export?
• Can the platform fit the team's current stack and release process without duplicating maintenance?
• Which capability, performance, customer outcome, pricing, or ROI claims can be independently verified through documentation or a controlled trial?

These questions apply to any platform. Answers that cannot be demonstrated on your own representative workflow should stay unverified until they are.

‍

Why Thunders is a strong option when you buy

If your evidence points to Buy or Hybrid, the next question is which platform removes the most maintenance rather than moving it around. Thunders is an AI test automation platform where AI agents write, run, and maintain end-to-end tests from plain-language descriptions, across web, mobile (native iOS and Android on real devices), API, and desktop applications. It was built around the cost categories in this article, and the mapping is direct.

On the governance side, Thunders is ISO 27001, SOC 2, and GDPR compliant, offers dedicated cloud, self-hosted, and on-premises deployment with your own AI model keys, and exports tests to Playwright on request, which covers most of the constraints that would otherwise push a team toward building.

The outcomes are on record. Agorapulse onboarded its whole product team in two hours and went from 0% to 80% test coverage in six weeks. Clarrio cut the time to write an end-to-end test from 40 minutes to under 10, doubled its release frequency, and has had zero production incidents since adopting Thunders. LegalPlace, with no prior automation experience, was operational in days and cut release validation from several sprints to one. Teams at Sopra Steria, PwC, and Infopro Digital also test with Thunders.

Where Thunders is not the answer: proprietary protocols, hardware-in-the-loop testing, and tests that depend on precise timing or performance measurement. Keep those in code and let Thunders cover the high-change, user-facing surface around them.

‍

Conclusion

Build vs buy test automation does not have a universal answer. The right operating model matches your team's change rate, specialised requirements, governance standards, risk tolerance, and capacity to own evidence over the life of the suite.

One principle holds across all three paths: measure maintenance burden over the full operating life, not only during implementation. A framework that costs little to build can cost far more to keep current as the application evolves. A platform that is quick to configure can create dependencies and switching costs that are expensive to exit.

If you want our direct answer to the title question: for a standard web product that ships weekly or faster, buying usually costs less to maintain, because the dominant cost is repair, and repair is exactly what intent-based AI agents take off the team. Build still wins on genuinely specialised ground.

Evaluate against a representative workflow, not a vendor demo, and collect evidence from your own history before committing. If your evidence points to Buy or Hybrid, run one of your real flows on Thunders and measure repair work against your baseline. Book a session with a Thunders expert.