TLDR
- A regression test checks that a change has not broken features that worked before.
- A regression affects behavior that worked before a change and no longer works after it.
- Regression testing, retesting, smoke testing and functional testing do not answer the same question.
- Four strategies decide which tests to rerun: the full suite, targeted selection, prioritization or impact analysis.
- See how they differ in the comparison table.
Regression testing raises a recurring question between developers and testers: what really sets it apart from other tests? The term non-regression testing adds to the confusion. Yet both terms describe the same practice.
The ambiguity is not just a matter of words. It shows up in how scenarios are chosen, how responsibilities are split and how tests are maintained. When nobody knows which risk each test covers, the reports end up losing their value.
An e-commerce cart example shows what a regression suite catches, and above all what it misses. Then comes telling regression tests apart from retests and smoke tests, deciding when to run them, and understanding why their results sometimes end up convincing no one.
What is regression testing?
Regression testing means rerunning existing tests after a change to check that what worked before still works.
Also known as non-regression testing.
Regression or non-regression testing: what's the difference?
Both terms describe the same practice. "Regression testing", the term used by the ISTQB and in English, names the defect you are trying to detect. "Non-regression testing", common in France and in European teams, names the expected outcome: the absence of regression. Same test, two points of view.
A regression is a step backward: a feature that used to work stops working after a software change. It is previously correct behavior that a change has degraded.
Regression testing is therefore not a test type in its own right. A unit test, an end-to-end test (E2E) or an API test becomes a regression test when you rerun it to check that a change has not broken what already existed.
This has nothing to do with linear or logistic regression in statistics.
Why regressions happen
A change sometimes breaks a feature without touching its code. Five causes come up again and again:
- Coupling: two features share a component. Changing the discount calculation can disrupt the shipping cost calculation.
- Shared state: several user flows use the same data. A cart and a payment module write to the same field.
- Implicit contracts: a component relies on behavior that was never documented. An API changes the order of its results and disrupts the service that consumes them.
- Configuration drift: the code stays the same, but the environment changes. A new cache setting causes stale data to be displayed.
- Dependency upgrades: a library changes its default behavior. An update changes how dates are validated in a form.
These dependencies and their effects are not necessarily visible when reviewing the modified code. That is precisely why teams rerun tests: looking at the diff is not enough to foresee every consequence of a change.
A concrete regression testing example
An e-commerce site already accepts promo codes. The team adds gift card payments. The tests run after this change produce the following results.
The existing regression test passed. Yet a defect remains in the cart: combining the two discount methods produces an incorrect amount.
The problem comes from shared state: both features modify the same cart field. No scenario checked how they interact.
All existing tests are green, yet the cart total is wrong. Counting passed tests tells you nothing about the combinations nobody thought of.
Regression testing vs retesting vs smoke testing vs functional testing: the differences
Checking what already exists, confirming a fix, validating something new: these activities do not share the same goal.
Smoke test or regression test? The first asks "does it start?", the second "does it still work?". A smoke test usually takes a few minutes; a full regression suite can run for several hours.
Retest or regression test? After a fix, the retest checks the corrected defect. Regression testing checks the other behaviors that may have been affected. The two are complementary.
Functional test or regression test? These notions are not mutually exclusive. A functional test can be rerun after a change to look for a regression.
Regression testing is not a separate test category: it describes the timing and the intent of the check.
When should you run regression tests?
Regression tests run after any change that could alter a software's behavior, even when the features involved were not directly modified.
Six events can trigger a regression run:
- A defect fix.
- A new feature.
- Code refactoring.
- A dependency upgrade.
- A configuration or environment change.
- A branch merge.
In your CI/CD pipeline, how often and how much you test depends on the stage:
- On every commit: unit tests and a subset of smoke tests.
- On every merge: the tests covering the impacted scope.
- Before production release: the full regression suite.
- Overnight: tests too long to run in the delivery pipeline.
This cadence fits within continuous testing in a CI/CD pipeline without forcing the full suite to run after every change.
In Agile, regression testing is not a phase saved for the end of the sprint: it is part of the Definition of Done. Release frequency sets the maximum acceptable test duration: a team that deploys three times a day cannot wait six hours to validate each release, however good its suite's coverage.
Who performs regression testing?
Responsibility for regression testing depends on the test level and the features involved:
- Developers write unit tests and integration tests.
- QA teams cover system behavior and user journeys.
- Product owners and business teams define and validate user acceptance testing (UAT) scenarios.
Who runs the regression tests? Once the suite is automated, the CI/CD pipeline triggers the tests. If the team still has to assign someone to launch the suite, its execution relies at least partly on manual steps.
Is regression testing part of user acceptance testing? Not exactly. UAT checks that a feature meets business expectations; regression testing checks that existing behavior still works. In practice, a UAT plan can include regression tests on critical flows, but the acceptance criteria remain distinct.
How much of the suite should you rerun?
Should you rerun every regression test after each change? Not necessarily. The right scope depends on the risk associated with the change and the time available before release.
This choice is reassessed with every change, based on its nature and the risk of regression.
Why teams stop trusting their regression suite
A regression suite does not become useless overnight. The runs continue, the reports pile up, then teams stop believing the results. Here is how trust erodes.
1. Flaky tests
A test fails one time in five even though the code has not changed. The team starts by rerunning it, then does so automatically, until nobody looks at its failures anymore. A single flaky test can be enough to cast doubt on the whole report.
2. Growing run time
With each release, new tests are added without the old ones being reassessed. The suite ends up exceeding the release window. The team then stops running it before every release and only runs it "when there's time".
3. Tests disabled "temporarily"
A test fails, blocks the pipeline and gets disabled until a fix arrives. A few sprints later, it still is. The number of skipped tests and how long they have been skipped then say more than the total number of tests.
4. Invisible maintenance
A UI change breaks tests meant to check business rules, simply because they depend on fragile selectors or flows. The team spends time trying to fix a broken test suite, without having planned for that workload. The bigger the suite, the heavier this maintenance becomes. Thunders' self-healing tests can reduce part of this work, but not eliminate it. You also need to measure what that maintenance actually costs.
5. Vanity coverage
The suite has 2,000 tests, but most of them cover secondary screens. The checkout flow, meanwhile, remains under-covered. The test count goes up without reducing the risk on the most critical features.
A suite the team no longer trusts can become worse than having no suite at all: it consumes time and resources while giving a false sense of security.
Five warning signs for a regression suite
Five indicators point to this decline:
- The flaky test rate exceeds a few percent: measure the share of runs whose result changes with no code change.
- The full suite no longer runs before every release: compare its actual run time with the available release window.
- Tests stay disabled for more than one sprint: track the number of skipped tests and the age of the oldest one.
- Defects reach production on flows that are supposedly covered: identify incidents where a test existed and had passed.
- Nobody knows exactly what the suite covers: check traceability between requirements, critical flows and test cases.
To interpret these signals, failure analysis helps distinguish real defects from flaky results.
Better 300 reliable tests on the flows that matter than 3,000 tests you have to rerun three times before you believe them.
Manual or automated regression testing?
Regression tests are particularly well suited to automation: they check known behavior and must be rerun after every software change.
Automation pays off most for repetitive, deterministic and frequent scenarios. A test that checks an order total calculation or an API response can run on every release, with no human intervention. That is why regression testing is among the first candidates for test automation.
Manual tests remain useful for exploring unexpected behavior, assessing visuals or usability, and flows whose UI changes every week. In these situations, maintaining an automated script can cost more than doing the check by hand.
Can you automate 100% of regression tests? No, and that is not the goal. The goal is to broaden coverage without proportionally increasing the volume of manual checks.
What solutions are there to automate regression testing?
When choosing a solution, the real question is not how many tests it lets you write. It is who will still be able to maintain them two years from now. Three families of tools coexist.
Open source frameworks: Selenium, Playwright, Cypress. Free and widely used, they run scripts written by developers. They give full control, but every test is code to maintain. That is where the invisible maintenance described above comes from.
Commercial automation platforms, such as Katalon or Tricentis Tosca. They add flow recording, reporting and reusable components for teams that do not want to code everything. Most still rely on selectors or object models that need updating every time the UI changes.
AI-driven testing platforms, such as Thunders. Thunders automates regression testing in natural language across web and mobile applications, APIs and SAP. A scenario is written like an instruction: "add a promo code and a gift card to the cart, check that the discount is applied only once". AI agents then run it by interpreting the intent on screen, without generating a script. This approach addresses three of the problems described above:
- Maintenance. Tests rely on intent rather than selectors, and self-heal when the UI changes.
- Flakiness. Failure analysis separates real defects from flaky results and creates the Jira ticket with the reproduction steps.
- Run time. Tests run in parallel on Chrome, Firefox, Safari and Edge, inside the pipeline (GitHub Actions, Jenkins, CircleCI, Azure DevOps).
Business teams can also write their own acceptance scenarios, and existing Selenium, Cypress or Playwright tests can be converted.
How to build regression tests that last
To build a reliable regression test suite, four steps help target the scenarios that matter:
- Start from real user flows, such as ordering or payment, rather than listing tests screen by screen.
- Automate the riskiest scenarios first: those that fail often or whose failure costs the most.
- Integrate tests into the CI/CD pipeline from day one, without waiting for the suite to be "complete".
- Budget for maintenance from the start: a suite is only useful if its tests stay reliable from release to release.
A useful suite delivers a credible result before every release. The guide explains how to automate your regression suite.
