What Is CI/CD in Automation Testing? API Tests as a Pipeline Gate

This is lesson 7.1 of API Testing in CI/CD, Regression and Performance. It explains what continuous testing in a pipeline means for API tests, and how a gate turns test results into a build decision. It helps to know what a Test Run Pack is: lesson 9.10 builds one.

What is CI/CD in automation testing?

CI/CD in automation testing means your automated tests run on their own every time the code changes, inside the pipeline that builds and ships that code, and a failed result stops the change from going any further. Nobody remembers to press Run: the pipeline does it, every time, and the build's pass or fail is the test result.

For API tests this matters more than most. An API is a contract other teams build on, so a renamed field or a new 500 breaks someone else's code. A pipeline that runs the API tests on every pull request finds that break while it's one commit old.

What is CI/CD in testing: three terms, one loop

The letters cover three practices, and each one changes when tests run:

Continuous testing is the thread through all three: the tests run at every stage, automatically, and their result decides whether the change moves on.

What a CI/CD pipeline in automation testing looks like

A pipeline is a list of stages. API tests appear in several of them, and each stage runs a different set for a different reason:

The rule of thumb: the earlier the stage, the faster and more focused the tests. A ten-minute suite on every push is fine; a two-hour one teaches people to skip the pipeline.

The step contract: exit code, results file, gate

Whatever CI tool you use, a test step talks to the pipeline in three ways.

1. The exit code. Zero means the step passed and the pipeline carries on. Anything else fails the step. This is the only signal most pipelines act on.
2. A results file. JUnit XML is the common format. GitHub Actions, Azure Pipelines, Jenkins and GitLab can all read it and show each test by name, so a failed build says which test failed.
3. A gate rule. Something has to turn "38 passed, 2 failed" into that exit code. The simplest rule is "any failure fails the build". A pass-rate threshold is the next step up: fail the build only when the share of passing tests drops below a number you choose.

A gate should also fail closed. If the test server can't be reached, or the run never finishes, the step must fail, never pass. A timeout that reports green is worse than no tests at all.

Worked example: a pass-rate gate on the sandbox store

Take a pack of ten tests against the sandbox store at https://demo.totalshiftleft.ai: list products, read the Wireless Mouse, read a user, read an order with ?expand=user,product, and six more. The pipeline gate is a 90 percent pass threshold.

• Ten of ten pass. 100 percent: the build goes on.
• Nine of ten pass. Exactly 90 percent is still at the threshold: the build goes on, and the JUnit report names the failing test for someone to look at.
• Eight of ten pass. 80 percent is below 90: the build fails.

Now simulate an outage. The first request below returns 200; the second asks the sandbox to answer 503, as if a dependency were down.

curl -X GET 'https://demo.totalshiftleft.ai/api/v1/products'

curl -X GET 'https://demo.totalshiftleft.ai/api/v1/products?error=503'

A test that expects 200 fails on the second call. One failure in four tests is a 75 percent pass rate, which fails a 90 percent gate.

Which threshold to choose? Start at 100 percent for a small, stable pack: any failure is news. Lower it only for a large pack where a known flaky test would otherwise block every build, and fix the flaky test rather than living with the lower number.

Shift left continuous testing: what moves earlier

Shift left continuous testing is the habit of running each check as early as it can run. In practice that means three moves:

• Contract checks move to the pull request. If the spec says price is a number, a test that asserts it runs before the merge, not in a release test phase.
• Tests run against the spec before the service is finished. A mock server built from the spec lets the consumer's tests run in CI while the provider is still in progress.
• Regression runs every night, not every release. A nightly pack finds the break the morning after, while the change is still fresh.

The later stages don't disappear. Staging regression and post-deploy smoke checks still catch what only a real environment shows, such as configuration and data. Shifting left adds earlier gates; it doesn't remove the last one.

Common mistakes

• One giant suite on every push. The pipeline gets slow, and people start merging around it. Split it into packs by stage.
• Tests that share one environment with no control over its data. Two pipelines running at once trip over each other's records. Give CI its own environment, or reset its data first.
• A gate that can't fail. A threshold of 0, a step marked "continue on error", or a timeout that counts as success: each turns the gate into decoration.
• No results file. A red build with no test names sends someone digging through logs.
• Secrets in the pipeline file. Credentials belong in the CI tool's secret store, never in the YAML.

In Shift-Left API: a pass-rate gate on a Test Run Pack

In Shift-Left API, the tests live in the product and your CI decides when they run. The unit a pipeline runs is a Test Run Pack. Packs, schedules and CI need the Professional or Enterprise edition, or the 15-day trial; the Free edition doesn't include them.

1. Build the pack. In Execution, create a Test Run Pack with the tests this stage should run. A pack runs against one environment, so a pipeline that tests dev and staging uses one pack for each. Copy the pack's ID.
2. Allow the API user. An administrator opens Settings, then Integrations, then Public API, and checks that the role of the user your pipeline signs in as is allowed.
3. Store the credentials. Put that user's email and password in your CI tool's secret store.
4. Add the step. There are three ready-made integrations: a GitHub Action, an Azure DevOps task and a Jenkins plugin. For GitLab CI, CircleCI, Bitbucket Pipelines or anything else, the pipeline calls the public REST API under /api/v1 instead. Set the pack ID and a pass threshold, for example 90.
5. Push. The step signs in, starts the pack, waits for that run to finish, and applies the gate.

The visible result: with every test passing, the step logs the decision PASSED. Break one test and the pass rate drops under 90, the step fails with Quality gate: GATE_FAIL_THRESHOLD, and the build stops. Either way the step writes shiftleft-test-pack-results.xml (JUnit) and a JSON summary into the workspace, and the run appears in Reporting like any other.

Three details are worth knowing. By default, any test that ends in ERROR, meaning it could not run to the end, fails the gate whatever the pass rate. A pack that is already running is refused, so the same pack never runs twice at once. And the run executes on the Shift-Left API server, not on your CI agent, so the server must be able to reach the API under test. Lesson 9.15 walks through the full pipeline demo, and the CI/CD integrations page lists every option.

Try it yourself

With a trial edition, import the sandbox spec from https://demo.totalshiftleft.ai/openapi/rest.json and make four read tests. On one of them, add the query parameter error=503 and keep its check for status 200. Put all four in a Test Run Pack and click Run Now. Read the pass rate, then decide what a 90 percent threshold and a 75 percent threshold would do to the build.

Recap

• CI/CD in automation testing means the pipeline runs your tests on every change, and their result decides whether the change moves on.
• What is CI/CD in testing for an API team: fast contract checks on each pull request, a regression pack before release, and a smoke pack after deploy.
• A CI/CD pipeline in automation testing needs one step with an exit code, a JUnit file and a gate; shift left continuous testing puts that step as early as it can run.

For the tool-neutral, stage-by-stage walkthrough, read API testing in CI/CD: a step-by-step pipeline guide. Next, in lesson 7.2, you run a pack from a GitHub Actions workflow and watch the gate pass and fail.