What Is API Regression Testing? Suites, Schedules and Triage

This is lesson 7.5 of API Testing in CI/CD, Regression and Performance. Lessons 7.1 to 7.4 put a test pack in a pipeline. This lesson is about what goes in that pack, and the run that happens when nobody pushes anything: the nightly regression.

What is API regression testing?

API regression testing re-runs the checks your API already passed, after every change, to prove the change didn't break anything that used to work. A regression is behaviour you accepted that has quietly changed: a status code, a field, a type, a value, a response time. The new feature works; the old endpoint next to it now returns 500. Regression tests are the ones that notice.

What counts as a regression in an API

A regression is any difference from behaviour you've already agreed to, which is more than "it errors":

The value row is the tricky one. Try it: read the Wireless Mouse, then ship a price change.

curl -X GET 'https://demo.totalshiftleft.ai/api/v1/products/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa'

curl -X PATCH 'https://demo.totalshiftleft.ai/api/v1/products/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' \
  -H 'Content-Type: application/json' \
  -d '{"price":29.99}'

A test that asserts price == 24.99 now fails. Whether that's a regression depends on intent. If marketing raised the price, the test is out of date: update it. If a currency bug multiplied it, file a bug. The test's job is to make someone decide; it can't decide for you. That's why good regression tests assert exact values only where the value is part of the contract, and types or ranges everywhere else.

Choosing the suite: regression testing for API projects

Running every test on every change is the simplest suite, and it stops scaling quickly. Most teams layer it:

1. Smoke: five to fifteen read-only checks, one per critical endpoint. Runs after every deploy, in a minute or two.
2. Pull-request subset: the tests for the endpoints and flows the change touches, plus the smoke set.
3. Full regression: every accepted test, including slow end-to-end flows and data-heavy cases. Runs on merge to main, or nightly.

Pick what joins the full suite by risk, not by count: endpoints that move money or personal data, endpoints other teams depend on, and every bug that ever reached production, as a test that would have caught it. A regression suite that grows from real failures is worth more than one generated to a coverage number.

Tags keep the layers manageable. Tag tests smoke, regression, orders, and let each pack select by tag, so a new test joins the right runs without anyone editing the pack.

An API regression testing framework: the six parts

Whatever tools you use, a regression setup that holds up has six parts:

1. The suite: accepted checks, each asserting one behaviour.
2. Selection: which tests run at which stage, ideally by tag.
3. A stable environment: known data, reset before the run, and not shared with someone's manual testing.
4. A trigger: the pipeline for merges, a schedule for nightly.
5. A baseline: last night's result, so "4 failures" reads as "2 new failures since yesterday".
6. Triage: a routine for the morning after. Is it the environment, the test data, an out-of-date test, or a real bug?

Most regression programmes fail at part 6. A nightly run that is red every morning for unclear reasons teaches the team to ignore it, and then it catches nothing.

Regression testing in API pipelines: flaky tests and false alarms

A flaky test passes and fails on the same code. It is the quickest way to lose trust in a regression run. Measure it by flip rate: how often a test's result changes from one run to the next. A test that flips in a third of its runs is not telling you anything about the API.

When a test turns flaky, quarantine it (run it, but out of the gate), find the cause (shared data, a timing assumption, a dependency's rate limit), fix it, and bring it back. Lowering the whole pack's pass threshold to absorb one flaky test hides the next real regression.

In Shift-Left API: a nightly regression pack

In Shift-Left API, a regression suite is a Test Run Pack on a schedule. Packs and schedules need the Professional or Enterprise edition, or the 15-day trial. Lesson 9.10 walks through the pack wizard click by click; here is the regression version.

1. Select by tag rule. In Execution, create a Functional pack. On step two, switch to A tag rule and include tests tagged regression. The rule is re-evaluated at every run, so a test tagged next week joins tonight's run automatically.
2. Notify on failure. On step four, turn on a notification for failed runs and pick one channel: email, Slack, Microsoft Teams or a webhook. Notifications are off by default, and each message links to that run's report.
3. Schedule it. On step five, set Repeat to Daily at 2:00 a.m., check the time zone, and schedule it. Upcoming Schedule shows the next run.

The scheduled run executes on the Shift-Left API server, so the server must be running at 2 a.m. In Reporting each scheduled run appears with the execution type Automated, next to the runs people started by hand.

The morning after a red run:

• Read the causes, not the rows. The run report groups failures into a handful of causes and gives each a verdict: the tests are out of date, a test-data problem, an environment problem, or a suspected bug in the API. Twenty failures with one cause are one problem. This analysis is in every edition, Free included.
• Look at the green runs too. Under Passed, but worth a look, the report flags tests that passed without really checking anything, and responses that no longer match the documented contract. A regression a passing suite wouldn't notice shows up here.
• Fix with review. With an AI provider configured, the report can propose fixes for tests that are out of date, which you review and apply. A suspected API bug is never "fixed" by editing the test.

Over weeks, Analytics → Project Analytics shows daily trends and the top failing tests with their common errors (Professional, Trial and Enterprise). On the trial and Enterprise, the Dashboards tab adds a Flaky Tests list: any test whose result flips in more than 30 percent of at least three runs, with its recent history.

Two practical rules. Give CI and the nightly schedule separate packs: a pack that is already running refuses a second start, so a push at 2:01 a.m. would fail its pipeline step. And when the API's spec changes, re-import it: Shift-Left API shows what changed for you to approve, and approving creates new endpoint versions rather than editing your tests, so the regression suite keeps checking the behaviour you agreed to. See the API regression testing page for the full picture.

Try it yourself

On a trial, import the sandbox spec from https://demo.totalshiftleft.ai/openapi/rest.json and add a few read tests. In Execution, build a pack with a tag rule, turn on a notification for failed runs on step four, set Repeat to Daily at 2:00 a.m. on step five, and schedule it. Check Upcoming Schedule now, and Reporting after the run.

Recap

• What is API regression testing: re-running accepted checks after every change, so behaviour you agreed to can't change unnoticed.
• Regression testing for API teams means layered suites (smoke, pull-request subset, full nightly), chosen by risk and selected by tag.
• An API regression testing framework needs a stable environment, a baseline and a triage routine as much as it needs tests.

For tools, reference architecture and a full checklist, read API regression testing: strategy, automation and CI/CD. Next, in lesson 7.6, you test whether the API stays fast under load.