What to Automate at Each CI/CD Stage: Tests and Gates

The stage-by-stage map

The difference between the smoke set and the regression set matters more than most teams expect; smoke testing vs regression testing explains why they must be separate suites.

The distinction matters because the two failure modes look very different. Teams that under-invest in CI/CD test automation release slowly and reactively, catching defects in production where they are most expensive. Teams that invest heavily but design poorly end up with slow, brittle pipelines that frustrate developers and get bypassed. Getting automated testing in CI/CD right requires understanding which tests belong at which stage, how quality gates work, and how to keep feedback loops tight enough to be useful. This aligns directly with the shift-left testing philosophy of catching defects as early as possible.

The rest of this guide covers the full picture: the lifecycle from commit to production monitoring, the tools that serve each stage, and the quality gate patterns that enforce standards without blocking productivity.

---

In this guide

1. What Is Automated Testing in CI/CD?
2. Why Stage-by-Stage Automation Matters
3. Key Components of CI/CD Test Automation
4. CI/CD Quality Gates: How to Block a Deploy on Failed Tests
5. The CI/CD Testing Pipeline Architecture
6. Tools by Pipeline Stage
7. CI/CD testing in practice: a worked example
8. CI/CD testing challenges and how to solve them
9. Best Practices for Automated Testing in CI/CD
10. CI/CD Test Automation Checklist
11. Frequently asked questions about CI/CD testing

Performance checks belong in the same pipeline and behave differently from functional ones: k6 vs JMeter vs Gatling compares the runners, and the k6 and JMeter walkthroughs cover thresholds, correlation and the CI wiring each one needs.

What Is Automated Testing in CI/CD?

Automated testing in CI/CD is the practice of running programmatic test suites automatically at defined stages of the continuous integration and continuous delivery pipeline. The core idea is that every code change triggers an automated validation process—tests run, results are evaluated, and the pipeline either proceeds to the next stage or fails with actionable feedback.

CI (Continuous Integration) refers to the practice of integrating code changes frequently and validating them automatically. CD (Continuous Delivery or Deployment) refers to the automated process of taking validated code through staging to production. Automated testing is the quality mechanism that makes both trustworthy.

Key characteristics of mature CI/CD test automation:

• Triggered automatically: Tests run on code events (push, PR, merge), not on manual schedule
• Stage-appropriate: Different test types run at different pipeline stages based on speed and scope
• Gate-enforcing: Failures stop the pipeline—they do not just produce warnings
• Results-visible: Test outcomes are visible in PR interfaces, dashboards, and notifications
• Maintained actively: Test suites are treated as production assets with defined ownership

The goal is a system where a developer receives comprehensive quality feedback within 5–10 minutes of submitting a change, enabling a tight loop between writing code and knowing whether that code works.

---

For the GitHub-specific detail — service containers, matrix sharding, OIDC and reusable workflows — see API testing in GitHub Actions.

On Microsoft tooling the equivalent is the Azure DevOps API testing pipeline guide, including the publish-test-results task most teams miss.

On Jenkins the equivalent setup is in how to run API tests in Jenkins pipelines, including the Jenkinsfile and JUnit publishing.

If the gap is tooling rather than process, start with continuous integration testing tools and its comparison matrix.

Why Stage-by-Stage Automation Matters

The Business Case

A defect is cheapest to fix when it is found close to the change that introduced it: the author still has the context, nothing has been built on top of it, and nobody outside the team has seen it. Every stage of automated validation shortens the distance between introduction and detection. The stage map exists to push each class of defect to the earliest stage that can catch it cheaply. Measure your own escaped-defect rate and change failure rate before and after, rather than relying on industry multipliers.

Developer Productivity

When a developer gets definitive feedback within minutes of opening a pull request, they fix the problem while the code is still in their head. When feedback takes hours or days, they have moved on, and the fix starts with re-reading their own change. That is why the pull-request stage has the tightest budget in the map: it is the stage developers wait on.

Regulatory and Compliance Requirements

In regulated industries (fintech, healthcare, aerospace, government), automated testing records in CI/CD pipelines serve as audit evidence. Demonstrating that every production deployment was preceded by automated validation of specific compliance-critical controls is increasingly a regulatory requirement, not just a best practice.

---

Key Components of CI/CD Test Automation

Component 1: Quality Gates

Quality gates are automated checkpoints that evaluate whether the pipeline should proceed or stop. Effective quality gates are:

• Objective: Defined by measurable thresholds (95% test pass rate, 80% code coverage)
• Enforced: Failure stops the pipeline, not just generates a warning
• Calibrated: Set at levels that catch real problems without generating excessive false positives
• Documented: Every gate has a stated rationale so engineers understand why it exists

Common quality gates by stage:

• Pre-commit: Linting pass, unit tests pass
• PR gate: All unit tests pass, all API tests pass, no new security vulnerabilities
• Build gate: Code coverage above threshold, integration tests pass
• Staging gate: Smoke tests pass, performance baseline within 10% of previous
• Release gate: Full regression pass, zero critical security findings

Component 2: Test Categorization by Speed

Categorizing tests by execution speed is essential for CI/CD efficiency:

• Fast tests (under 100ms each): Unit tests, isolated component tests
• Medium tests (100ms–2s each): API tests, integration tests with mocked dependencies
• Slow tests (2s+): E2E browser tests, load tests, tests requiring external services

Fast tests run on every commit. Medium tests run on every build. Slow tests run on a schedule. This categorization keeps feedback loops tight where they matter most—at the commit and PR stages where developers can act on feedback immediately.

Component 3: Parallel Execution Infrastructure

Sequential test execution cannot scale to modern delivery cadences. Parallel execution distributes test load across multiple runners, reducing wall-clock time proportionally to the number of runners. A test suite that takes 20 minutes sequentially can run in 4 minutes across 5 parallel runners.

Requirements for effective parallel execution:

• Tests must be completely independent (no shared state)
• Test data must be created and cleaned up per test
• Configuration must support multiple simultaneous instances
• Reporting must aggregate results from all parallel runners

Component 4: Feedback and Notification

Test results need to reach the right people at the right time:

• PR interface annotations: Show test status directly in the GitHub/GitLab PR view
• Slack/Teams notifications: Alert the responsible engineer on failure
• Dashboard: Aggregate test health metrics for team review
• Trend alerts: Notify when execution time or failure rates change significantly

Component 5: API Testing Integration

API testing at the build stage is the highest-value CI/CD testing investment for most teams. APIs are the interface between services, the implementation of business logic, and the contract that UI components depend on. Continuous validation of APIs in CI prevents the class of defects that slip through unit tests (which test in isolation) and only surface during integration.

Shift-Left API integrates directly into CI/CD pipelines by importing OpenAPI/Swagger specifications and generating complete API test suites that run automatically on every build through first-party GitHub Actions, Jenkins and Azure DevOps integrations, with GitLab CI and other tools calling the same public REST API. See also: API test automation with CI/CD step-by-step guide.

How Shift-Left API does this: Shift-Left API generates the API test suite directly from an imported OpenAPI or Swagger spec, then runs a test pack as a pipeline gate through its CI runner, which writes JUnit XML and a JSON summary and fails the build below a configurable pass-rate threshold. The runner ships as an official GitHub Action, a Jenkins plugin and an Azure DevOps task; the GitLab CI component, CircleCI orb and Bitbucket pipe are public in source but not yet published to their registries, so those platforms — and any other CI/CD tool — use the public REST API today.

---

CI/CD Quality Gates: How to Block a Deploy on Failed Tests

A quality gate is only real if it can actually stop a merge or a deploy — a check that reports red but doesn't block anything trains the team to ignore it within weeks. Blocking works differently depending on where the gate sits:

At the PR gate, use required status checks. In GitHub, that means every job in your workflow that should be able to block a merge needs a name, and that name gets added to branch protection:

# .github/workflows/ci.yml
name: CI

on:
  pull_request:
    branches: [main]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test -- --coverage

  api-tests:
    runs-on: ubuntu-latest
    needs: unit-tests
    steps:
      - uses: actions/checkout@v4
      - name: Run API test pack
        uses: Total-Shift-Left/Shift-Left-API-Integrations/github-actions@v1
        with:
          server-url: ${{ secrets.SHIFTLEFT_URL }}
          api-email: ${{ secrets.SHIFTLEFT_EMAIL }}
          api-password: ${{ secrets.SHIFTLEFT_PASSWORD }}
          pack-id: ${{ vars.SHIFTLEFT_TEST_PACK_ID }}
          pass-threshold-percent: '100'   # the gate; writes shiftleft-test-pack-results.xml (JUnit)

Then in the repository's Settings → Branches → Branch protection rules, enable "Require status checks to pass before merging" and select unit-tests and api-tests explicitly. Without this step, the jobs run and report status but a PR can still be merged with them red.

In GitLab CI, gating is implicit: any job that exits non-zero fails the pipeline, and merge request approval rules can require the pipeline to succeed:

# .gitlab-ci.yml
stages: [test]

api-tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache bash curl jq
  script:
    # ci/run-shiftleft-pack.sh is the script from the REST API guide (linked below):
    # it signs in, notes the pack's latest run, starts the pack, waits for the NEW run
    # to finish, and exits non-zero when that run's pass rate is below the threshold.
    - SHIFTLEFT_PACK_ID="$SHIFTLEFT_TEST_PACK_ID" SHIFTLEFT_PASS_THRESHOLD=95 bash ci/run-shiftleft-pack.sh

The script is in Run a pack from any CI with the REST API. Polling the status endpoint straight after starting a run can read the previous run and pass the build, so the script waits until the run id changes before it grades anything.

Set Settings → Merge requests → "Pipelines must succeed" to make this binding rather than advisory.

At the pre-production gate, the pattern is the same but the target environment and threshold usually change — a PR gate might require 100% pass on a smoke subset, while a pre-production gate requires the full regression suite above a pass-rate threshold rather than 100% (flaky third-party dependencies make 100% unrealistic at that stage for most teams). Configure the threshold explicitly rather than defaulting to "no failures," and document why that threshold was chosen — an undocumented gate gets disputed the first time it blocks someone.

The CI/CD Testing Pipeline Architecture

---

Tools by Pipeline Stage

---

CI/CD testing in practice: a worked example

Illustrative scenario: a composite rollout, not a named customer's measured results.

Problem

A B2B SaaS team of about 18 engineers had a CI/CD pipeline for build and deployment, but no automated testing in it. Two QA engineers tested manually after each build, releases went out every three weeks, and each release took two days of manual QA.

Solution

The team implemented automated testing across all pipeline stages over 10 weeks.

Phase 1: Commit and PR gates (weeks 1–3)

• Installed Husky for pre-commit hooks: linting + fast unit test runs
• Configured GitHub Actions with three parallel runners on every PR:
  • Runner 1: Full Jest unit test suite with coverage report
  • Runner 2: Shift-Left API API tests against the development environment
  • Runner 3: SonarQube static analysis
• Configured required status checks: all three runners must pass before merge

Phase 2: Build stage validation (weeks 4–6)

• Added TestContainers to run integration tests with real PostgreSQL and Redis
• Implemented Pact for contract testing between frontend and API
• Set coverage gate: PRs reducing coverage below 75% are blocked

Phase 3: Staging validation (weeks 7–8)

• Smoke test suite using Playwright (15 critical user paths)
• Shift-Left API re-runs API tests against the staging environment post-deployment
• Performance baseline with k6: any endpoint exceeding P95 > 500ms is flagged

Phase 4: Production monitoring (weeks 9–10)

• Configured Datadog Synthetics to probe 12 critical API endpoints every minute
• Set up PagerDuty alerting on error rate spikes above 1%
• Weekly DORA metrics review added to engineering retrospective

What changes

In a rollout like this, pull-request feedback moves from days to minutes, manual QA shrinks to exploratory testing, and releases can go out more often because each one is no longer a two-day event. Track deployment frequency, change failure rate, time to restore and PR feedback time against your own starting point; those four numbers show whether the stage map is working.

---

CI/CD testing challenges and how to solve them

Challenge: CI pipeline takes 45+ minutes Solution: Audit each stage for unnecessary sequential dependencies. Parallelize wherever possible. Move slow tests to scheduled nightly runs rather than every-commit gates. Use caching aggressively (npm cache, Docker layer cache, dependency cache).

Challenge: Tests pass in CI but fail in staging Solution: This indicates environment-dependent tests. Audit for hardcoded URLs, environment-specific data assumptions, and untested configuration differences. Use environment variables consistently and ensure CI and staging environments are configured from the same source.

The same applies to performance. Rather than a separate k6 script that drifts from the functional suite, Shift-Left API's performance and load testing builds load scenarios from the functional tests you already run — same auth, headers and bodies — can be started from CI with an API key, and compares each run to a pinned baseline with a statistical test over p95. It is a paid add-on to Professional, Custom or Enterprise licences, included in the Trial; k6 and Gatling remain good choices when you want load scripts as code.

Challenge: API tests require manual setup every time Solution: Use Shift-Left API, which maintains API test suites from your OpenAPI specification. When endpoints change, re-import the spec and regenerate: the generator reports which tests it created, updated or skipped and why, and any fix to an existing test is a proposal a person accepts rather than a silent rewrite.

Challenge: Developers disable failing tests to unblock deploys Solution: Establish a policy that only quarantined tests (tracked in a dedicated backlog) can be bypassed. Disabling tests without quarantine tracking is treated as a process violation. Make quarantined test count a visible metric.

Challenge: No one knows what the CI failures mean Solution: Invest in failure reporting. Every CI failure should link directly to the failing test, the error message, and a reproduction command. Good reporting reduces MTTD (Mean Time to Diagnose) from hours to minutes. Grouping helps more than detail: Shift-Left API's failed-run analysis turns a red run into a handful of causes, each with a verdict — test defect, data issue, environment problem or suspected product bug — so thirty failures caused by one expired sign-in read as one problem, not thirty.

Challenge: Security scans block every build with false positives Solution: Configure your SAST tool to suppress known false positives with documented rationale. Establish a triage process for new findings. Block only on critical and high severity findings with high confidence scores.

---

Best Practices for Automated Testing in CI/CD

• Run tests at every pipeline stage. Every stage without automated tests is a blind spot in your quality coverage.
• Fail fast and specifically. The error message from a failed quality gate must tell a developer exactly what broke and why.
• Keep PR feedback under 10 minutes. Developers cannot maintain context on PRs that take longer to validate. Use parallelism to enforce this budget. For the API stage in detail, see the API testing in CI/CD guide.
• Use Shift-Left API for API coverage in CI. Auto-generation from OpenAPI specs eliminates the gap between API documentation and API validation.
• Never allow failed tests to deploy. Quality gates must gate, not advise.
• Track DORA metrics and connect them to test automation maturity. Deployment frequency, change failure rate, and MTTR all improve directly as CI/CD test automation matures. A comprehensive DevOps testing strategy connects these metrics to your overall quality approach.
• Make test results visible to everyone. Test dashboards, Slack notifications, and PR annotations keep quality front-of-mind for the whole team.
• Treat CI configuration as code. Pipeline definitions belong in version control, reviewed like application code.
• Follow established test automation best practices for DevOps. Deterministic tests, proper test data management, and parallel execution are foundational to reliable CI/CD test automation.
• Test your CI pipeline. Pipeline failures should trigger alerts just like test failures. A broken CI system is worse than no CI system.
• Continuously prune slow and irrelevant tests. Every quarter, review the test suite for tests that have not caught a defect in 6+ months. Delete or refactor them.

---

CI/CD Test Automation Checklist

• ✔ Pre-commit hook runs linting and fast unit tests (under 60 seconds)
• ✔ PR gate runs full unit and API tests in parallel (under 10 minutes)
• ✔ Build stage runs integration tests and enforces coverage threshold
• ✔ Staging deployment triggers smoke tests and API validation automatically
• ✔ Quality gates block deployment on failure—no exceptions without documented quarantine
• ✔ API endpoints are continuously tested via TSL integration (all endpoints covered)
• ✔ Test results are posted to PR interface and team notification channel
• ✔ DORA metrics (deployment frequency, CFR, MTTR) are reviewed monthly

---

See also: contract testing in our learn hub for the underlying concept.

Related reading: Continuous Testing vs Shift Left Testing: Key Differences

Frequently asked questions about CI/CD testing

What should you automate at each CI/CD stage?

Pre-commit: lint and fast unit tests. Pull request: the full unit suite, API and contract tests, and static security analysis. Merge to main: integration tests and the full regression suite. After each deployment: a smoke set. Nightly or pre-release: end-to-end journeys, load and deep security scans. Production: synthetic monitoring.

What tests should run in a CI/CD pipeline?

Unit tests run on every commit. API and contract tests run on every pull request. Integration and the full regression suite run on merge to main. Smoke tests run after each deployment. Long end-to-end and load tests run nightly or before a release.

Why are quality gates important in CI/CD?

Quality gates are automated checkpoints in the CI/CD pipeline that stop the delivery process if predefined quality thresholds are not met. Common quality gates include test pass rate thresholds, code coverage minimums, and security scan clean bills of health. They prevent substandard code from reaching production.

How do you automate API testing in CI/CD without writing code?

Shift-Left API imports your OpenAPI/Swagger specification and auto-generates API tests that connect directly to your CI/CD pipeline via first-party GitHub Actions, Jenkins or Azure DevOps integrations, or its public REST API from GitLab CI and other tools. No custom test code is required.

---

Sources and further reading

• GitHub Actions documentation — workflow syntax, caching and matrix builds.
• GitLab CI/CD documentation — .gitlab-ci.yml stages, rules and artifacts.
• DORA State of DevOps research — the multi-year dataset behind the four delivery metrics.

Key takeaways

Automated testing in CI/CD is the mechanism that separates teams that ship with confidence from teams that ship and hope. By implementing stage-appropriate testing—from pre-commit unit tests through production synthetic monitoring—and enforcing quality gates at every stage, teams can achieve the deployment frequencies and change failure rates that define elite software delivery performance. The API layer is your most valuable CI/CD testing investment, and Shift-Left API makes it accessible without writing a line of test code. Start your free trial and bring automated API testing to your CI/CD pipeline today.

---

Related: API Testing in CI/CD | Smoke Testing vs Regression Testing | API Quality Gates | What Is Shift Left Testing | CI/CD Integrations | Start Free Trial