Smoke Testing vs Regression Testing: Key Differences

Teams that blur the two get one of two bad outcomes. Some run the full regression suite after every deployment and spend half an hour testing an environment that can't sign anyone in. Others run only a smoke check before a release and ship a regression from three sprints ago.

---

In this guide

1. Smoke vs regression testing at a glance
2. What is smoke testing?
3. What is regression testing?
4. Smoke vs sanity vs regression testing
5. When to choose smoke testing and when to choose regression testing
6. Where each runs in a CI/CD pipeline
7. How to automate both suites for an API
8. Common mistakes and how to fix them
9. Smoke testing vs regression testing FAQ

Smoke vs regression testing at a glance

The verdict: smoke testing is a deployment check and regression testing is a change check. Run smoke right after every deployment and stop everything if it fails. Run regression on every pull request (the fast API set) and before every release (the full suite), and block the merge or the release if it fails. Neither replaces the other.

---

What is smoke testing?

Smoke testing, also called build verification testing, is a minimal set of checks that a build or deployment works at a basic level. The name comes from hardware: power on the board, and if nothing smokes, it's worth testing further. A smoke suite is deliberately small. It doesn't try to be thorough. Its only job is to decide whether investing in more testing makes sense.

What an API smoke suite checks:

• The service is reachable and the health endpoint returns 200.
• Authentication is wired up: a protected endpoint returns 401 without credentials and 200 with them.
• The database answers: a cheap read returns data.
• The one or two most critical paths work end to end (for example, place an order, or fetch an account).
• Dependencies the service can't run without are reachable.

What it deliberately doesn't check: business-rule correctness, edge cases, every endpoint, performance under load, or whether yesterday's behaviour still holds.

If any of these fail, the deployment itself is broken. Stop, notify whoever owns the deployment, and don't move on to regression.

---

What is regression testing?

Regression testing re-runs an established suite after a change to prove that previously working behaviour still works. A regression is a step backwards: something that worked before and doesn't now. The suite covers every endpoint and the scenarios that matter on each one: success paths, error handling for invalid input, authorization rules, schema and contract checks, and tests for bugs that were fixed before, so they can't quietly come back.

For APIs, regression testing is fast enough to run on every pull request. A spec-driven API suite covering hundreds of operations usually finishes in minutes. An end-to-end UI suite with the same functional scope can take an hour, so that one belongs to nightly or pre-release runs. That's why most teams split regression into a fast API set that gates pull requests and a full sweep that gates releases.

Regression has a performance cousin too. A load run answers a different question from a functional regression run: did it get slower, or fall over under load? Some load tools offer a "smoke load" profile, a short light run that confirms the system holds up at all before a longer load or soak run is worth starting. Shift-Left API's performance and load testing add-on (paid, included in the Trial) builds these runs from the functional tests you already have.

---

Smoke vs sanity vs regression testing

Sanity testing is the third term people mix up with the other two. The working definitions:

Some teams use "sanity" and "smoke" interchangeably, which is fine as long as the team agrees. What matters is that the post-deploy check stays small and the regression suite stays separate.

---

When to choose smoke testing and when to choose regression testing

Choose a smoke run when:

• A deployment has just finished, to any environment, including production.
• You need a go or no-go decision in minutes, for example before handing an environment to other teams or starting a long test run.
• You're promoting the same artifact through environments and need to know the new environment's configuration is sound.
• An incident fix is going out and you need fast confirmation that the service is up.

Choose a regression run when:

• A pull request changes behaviour, a dependency or a schema. Run the fast API regression set as the merge gate.
• A branch merges to main or a release is being cut. Run the full suite, including the slower journeys.
• The API spec changed. Run regression plus a breaking-change check on the spec itself.
• A bug was fixed. Add a test that would have caught it, and let regression keep it fixed.

Choose both, in order, when you deploy a release candidate to staging: smoke first (is it alive?), then regression against staging (did anything break?), then promote.

---

Where each runs in a CI/CD pipeline

A typical ordering:

1. Pull request: unit tests, then the fast API regression set against a test environment. The regression set is the merge gate.
2. Merge to main: full API regression, integration and workflow tests.
3. Deploy to staging: smoke set (under two minutes), then API regression against staging.
4. Pre-release or nightly: end-to-end journeys and load runs.
5. Deploy to production: smoke set straight away, then synthetic monitoring.

Each stage has a gate. A failed smoke run stops promotion and triggers rollback or a fix to the deployment. A failed regression run blocks the merge or the release. Making those gates actually block, rather than just turning red, is covered in API quality gates: what to measure. The full stage map, including the non-API layers, is in what to automate at each CI/CD stage.

---

How to automate both suites for an API

A smoke suite in a shell script. It should finish in seconds, so plain curl is enough:

#!/usr/bin/env bash
# smoke.sh: runs right after a deploy and finishes in about 20 seconds
set -euo pipefail
API=${1:?}
check() { [ "$(curl -so /dev/null -w '%{http_code}' "$2")" = "$1" ] || { echo "FAIL $2"; exit 1; }; }

check 200 "$API/health"
check 401 "$API/v1/orders"                       # auth is wired up
check 200 "$API/v1/catalog?limit=1"              # the database answers
echo "smoke passed"

Both suites in one codebase, split by marker. With pytest, a marker makes the split explicit and enforceable:

# pytest.ini
# [pytest]
# markers =
#     smoke: deploy gate, must finish in under 60s
#     regression: full behavioural sweep

@pytest.mark.smoke
def test_service_is_up(api):
    assert api.get("/health").status_code == 200

@pytest.mark.regression
@pytest.mark.parametrize("qty,expected", [(1, 201), (0, 422), (-1, 422), (10_001, 422)])
def test_order_quantity_boundaries(api, qty, expected):
    assert api.post("/v1/orders", json={"sku": "A-1", "qty": qty}).status_code == expected

# pytest -m smoke        -> after every deploy
# pytest -m regression   -> on every pull request

The same split without code. In Shift-Left API, tag tests smoke or regression and build a test pack from each tag rule. The rule is re-evaluated at every run, so newly tagged tests join automatically. Point the post-deploy step at the smoke pack and the pull-request gate at the regression pack. The CI runner writes JUnit XML and fails the build when a pack's pass rate drops below its threshold, so each suite keeps its own gate. A first-party GitHub Action, Jenkins plugin and Azure DevOps task are available, and any other CI calls the REST API; see automated API regression testing.

When a run fails, it helps to know whether you're looking at a broken deployment or a real regression. Shift-Left API groups a failed run into causes, each with a verdict (test defect, data issue, environment problem or suspected product bug). A deployment that returns a gateway error page then shows up as one environment problem instead of forty failing tests, which is exactly the signal a smoke stage exists to give.

---

Common mistakes and how to fix them

The smoke suite keeps growing. Every incident adds "just one more check" until smoke takes ten minutes. Fix: cap it at around 30 tests and two minutes. Anything that isn't required for further testing to be meaningful belongs in regression.

The full regression suite runs after every deployment. It wastes time on broken environments and slows down rollback decisions. Fix: smoke first, then regression only if smoke passes.

Regression catches things smoke should have caught. A missing secret surfaces as 300 regression failures. Fix: after each such run, ask whether a minimal smoke check would have caught it, and add one if so.

The regression suite grows without bound. Fix: for API regression, generate the baseline from the OpenAPI spec so its size follows the spec, not habit. For end-to-end regression, keep a count budget and delete redundant tests every quarter.

Nobody knows what belongs in smoke. Fix: look at the last six months of deployment failures. The causes (unreachable service, broken sign-in, failed migration, dead dependency) are your smoke checklist.

---

Smoke testing vs regression testing FAQ

What is the difference between smoke testing and regression testing?

Smoke testing asks whether a build is alive and worth testing further. It's a handful of critical-path checks that finish in a minute or two. Regression testing asks whether anything that used to work has broken. It's the large suite covering every feature, error path and business rule, and it takes longer.

Is smoke testing part of regression testing?

No. Smoke tests are often drawn from the regression suite, but they're a separate, much smaller suite with a different job and trigger. Smoke runs after every deployment and stops everything if it fails. Regression runs on pull requests and before releases, and it blocks the merge or the release.

Which comes first, smoke testing or regression testing?

Smoke testing comes first after a deployment. There's no point spending twenty minutes on a regression run against an environment that can't sign a user in. On pull requests, the regression set runs against the test environment as the merge gate.

What is the difference between smoke, sanity and regression testing?

Smoke testing checks that a new build is alive across its critical paths. Sanity testing is a narrow, focused check that a specific fix or change works, usually before a deeper run. Regression testing checks that everything that worked before still works after the change.

Can smoke testing be automated?

Yes, and it should be. An API smoke suite is a few health, sign-in and critical-path calls that a pipeline runs right after each deployment, and a non-zero exit stops promotion. Manual smoke checks get skipped exactly when a deployment is rushed, which is when they matter most.

How many tests should a smoke suite have?

Usually 10 to 30, finishing in under two minutes. If a smoke suite grows past that, it's turning into a regression suite. Move the extra tests to regression and keep smoke to the paths that must work for any further testing to be meaningful.

---

Sources and further reading

• ISTQB Glossary: the standard definitions of smoke, sanity and regression testing.
• Martin Fowler, The Practical Test Pyramid: the reference argument for test-layer proportions.
• Google Testing Blog: Google's engineering write-ups on test sizing and flakiness.
• pytest: working with custom markers: how to split one codebase into smoke and regression runs.

Key takeaways

• Smoke testing is a deployment check. Regression testing is a change check. Keep them as separate suites.
• Run smoke right after every deployment and stop everything if it fails. Run the fast API regression set on every pull request and the full suite before release.
• Keep smoke to about 30 tests and two minutes. Anything bigger is regression in disguise.
• Automate both, and give each suite its own gate in the pipeline.

To run both suites from one place, with tests generated from your OpenAPI spec, separate smoke and regression packs, and a pass-rate gate for each in your CI, start a free 15-day trial.