API Versioning Testing: Backward Compatibility Checks in CI

It is the backward-compatibility branch of OpenAPI testing: the spec is the contract, and versioning tests check that the contract only changes on purpose. If you are new to the idea, read what API contract testing is first; this guide assumes you have an OpenAPI document under version control.

API versioning strategies and what each one means for tests

There are four common ways to say which version a client wants. Each changes what your tests must send and assert.

Whichever you choose, the most common versioning bug is a silent fallback: a client asks for a version that does not exist and quietly gets the latest one. Write a test that asks for an unsupported version and expects a clear 400, 404 or 406 with a message naming the supported versions.

Breaking vs non-breaking changes

Backward compatibility is defined by consumers: a change is breaking if a client that worked yesterday can fail today without changing its code.

The last row matters because no spec diff will catch it. Defaults and behaviour are only protected by tests.

Step 1: Diff the spec against the last release

A spec diff is the cheapest versioning test you can run: it needs no running service and finishes in seconds. Two well-maintained open-source tools do it.

oasdiff classifies every change as error, warning or info and exits non-zero when you ask it to:

# base.yaml = spec from the last release, openapi.yaml = this branch
docker run --rm -v "$PWD:/specs" tufin/oasdiff \
  breaking /specs/base.yaml /specs/openapi.yaml --fail-on ERR

OpenAPITools openapi-diff (Java) produces a readable report and fails on incompatible changes:

docker run --rm -v "$PWD:/specs:ro" openapitools/openapi-diff:latest \
  /specs/base.yaml /specs/openapi.yaml --fail-on-incompatible

As a CI gate on every pull request:

# .github/workflows/api-compat.yml
name: api-compat
on: [pull_request]

jobs:
  breaking-changes:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Get the released spec
        run: git show origin/main:openapi.yaml > base.yaml
      - name: Fail on breaking changes
        run: >
          docker run --rm -v "$PWD:/specs" tufin/oasdiff
          breaking /specs/base.yaml /specs/openapi.yaml --fail-on ERR

Compare against the released spec (a tag, or main if main is what you deploy), not the previous commit, or a breaking change split across two pull requests slips through. When a break is intentional, the fix is not to silence the gate: add a new version and leave the old one untouched.

The diff only proves the document is compatible. To prove the running service matches the document, validate live responses against it; our guide to catching schema drift covers that half. Shift-Left API's API contract testing does the diff when you re-import a spec: it classifies each change as breaking or non-breaking, holds changes for review unless you opt in to auto-approving non-breaking ones, and keeps approved changes as new endpoint versions so the old definition stays in history. Start a free trial to run that check on your own spec.

Step 2: Run a suite for every supported version

A spec diff will not notice that a shared service method now rounds prices differently in v1. Tests will. Keep one suite per supported version, each validating against a frozen copy of that version's schema, and run them all on every build.

# tests/test_versions.py  (pip install pytest requests jsonschema)
import json
import os
import pathlib

import pytest
import requests
from jsonschema import Draft202012Validator

BASE = os.environ.get("API_BASE_URL", "http://localhost:8080")
SCHEMAS = pathlib.Path(__file__).parent / "schemas"
SUPPORTED = ["v1", "v2"]


def schema_for(version):
    return json.loads((SCHEMAS / f"order.{version}.json").read_text())


@pytest.mark.parametrize("version", SUPPORTED)
def test_order_matches_its_frozen_schema(version):
    r = requests.get(f"{BASE}/{version}/orders/42", timeout=10)

    assert r.status_code == 200
    errors = list(Draft202012Validator(schema_for(version)).iter_errors(r.json()))
    assert not errors, "\n".join(f"{list(e.path)}: {e.message}" for e in errors)


def test_unknown_version_is_rejected_not_redirected():
    r = requests.get(f"{BASE}/v9/orders/42", timeout=10)

    assert r.status_code == 404


def test_v1_and_v2_agree_on_shared_fields():
    v1 = requests.get(f"{BASE}/v1/orders/42", timeout=10).json()
    v2 = requests.get(f"{BASE}/v2/orders/42", timeout=10).json()

    assert v1["id"] == v2["id"]
    assert v1["total"] == v2["amount"]["value"]  # v2 renamed and nested the field

The frozen schema (order.v1.json) is copied out of the spec when v1 is released and never edited again. If a later change makes the v1 test fail, the change broke v1, whatever the current spec says.

For header or media-type versioning, parametrize the header instead of the path:

@pytest.mark.parametrize("accept, version", [
    ("application/vnd.example.order.v1+json", "v1"),
    ("application/vnd.example.order.v2+json", "v2"),
])
def test_media_type_versions(accept, version):
    r = requests.get(f"{BASE}/orders/42", headers={"Accept": accept}, timeout=10)

    assert r.status_code == 200
    assert r.headers["Content-Type"].startswith(accept)


def test_unknown_media_type_version_returns_406():
    r = requests.get(f"{BASE}/orders/42",
                     headers={"Accept": "application/vnd.example.order.v9+json"},
                     timeout=10)

    assert r.status_code == 406

Step 3: Verify consumer contracts

Spec diffs and per-version suites protect what you documented. Consumers depend on what they actually use, which can be narrower (they ignore half the fields) or wider (they rely on an undocumented sort order). Consumer-driven contracts capture that.

With Pact, each consumer publishes the interactions it relies on, the provider verifies them in its pipeline, and a release is allowed only if every consumer's contract passes:

pact-broker can-i-deploy \
  --pacticipant orders-api \
  --version "$GIT_SHA" \
  --to-environment production

This is the strongest backward-compatibility signal you can get, because it is defined by the consumers themselves. It also tells you when an old version is safe to retire: when no consumer contract exercises it any more. For a comparison of contract approaches and tools, see contract testing for microservices.

Step 4: Test deprecation and sunset

Retiring a version is a feature, and it deserves tests. Two standard response headers announce it:

• Deprecation (RFC 9745) says the resource is or will be deprecated, as a structured date such as Deprecation: @1767225600.
• Sunset (RFC 8594) gives the HTTP date after which it may stop responding.
• A Link header can point to the replacement (rel="successor-version") and to migration docs (rel="deprecation").

def test_v1_announces_deprecation_and_sunset():
    r = requests.get(f"{BASE}/v1/orders/42", timeout=10)

    assert r.headers.get("Deprecation", "").startswith("@")
    assert "Sunset" in r.headers
    assert 'rel="successor-version"' in r.headers.get("Link", "")


def test_v2_is_not_marked_deprecated():
    r = requests.get(f"{BASE}/v2/orders/42", timeout=10)

    assert "Deprecation" not in r.headers

The second test is the one teams forget: a shared middleware that stamps deprecation headers on every route will happily mark the new version as deprecated too.

Common mistakes in API versioning testing

• Only testing the latest version. Shared code paths mean a v2 change can break v1. Every supported version stays in the matrix until its sunset.
• Diffing against the previous commit. Diff against the released spec, or breaking changes split across pull requests slip through.
• Editing old schemas to make tests pass. A frozen schema that changes is not frozen. If v1's test fails, fix the code or ship v3.
• Ignoring behavioural changes. Default page size, sort order, rounding and error format are part of the contract even though the spec does not show them.
• Bumping the version for every change. Additive changes do not need a new version; they need a diff gate proving they are additive. Frequent major versions multiply the test matrix and consumer pain.
• No test for unsupported versions. Silent fallback to the latest version is the most common versioning bug.

API versioning testing FAQ

What is API versioning testing?

It checks that each version of an API you still support keeps its contract while new versions ship: a spec diff for breaking changes, a suite per version against that version's frozen schema, consumer contract verification, and checks on unknown, default and deprecated versions.

How do you test backward compatibility of an API?

Diff the new OpenAPI document against the last released one with oasdiff or openapi-diff and fail on breaking changes. Then run the previous version's suite, unchanged, against the new build and verify every consumer contract.

Is it better to version an API in the URL or in a header?

URL versioning is the easiest to see, route, cache and test. Header or media-type versioning keeps URLs stable but is harder to debug and needs Vary for caches. Either works if the rule is consistent and missing or unknown versions get a documented response.

Is adding a new enum value a breaking change?

Not in a request. In a response it often is, because clients that deserialize enums strictly will fail on an unknown value. Treat it as breaking unless your contract tells consumers to tolerate unknown values.

What tools detect breaking changes in OpenAPI specs?

oasdiff and OpenAPITools openapi-diff compare two OpenAPI documents and exit non-zero on breaking changes, so both work as CI gates. Pact's can-i-deploy covers the consumer-driven side.

How long should old API versions be tested?

For as long as they are supported: until the Sunset date has passed and traffic shows consumers have moved.

Sources and further reading

• oasdiff documentation and breaking-change rules
• OpenAPITools openapi-diff
• OpenAPI Specification
• RFC 9745: The Deprecation HTTP Response Header Field
• RFC 8594: The Sunset HTTP Header Field
• RFC 9110: HTTP Semantics, content negotiation and Vary
• Pact docs: can-i-deploy
• Microsoft: Azure REST API guidelines on versioning
• Martin Fowler: Tolerant Reader