Contract Testing: Catching Breakage Before Clients See It

What a contract is

A contract is the published agreement between an API producer and its consumers: what endpoints exist, what inputs they take, what outputs they return, and under what conditions.

Contracts live in concrete artifacts:

• REST: OpenAPI / Swagger specification.
• GraphQL: SDL schema.
• SOAP: WSDL + XSD.
• gRPC: .proto files.
• Async: AsyncAPI for events.

Contract testing verifies that the implementation matches the contract. The contract is the source of truth; the implementation should never silently drift from it.

Three flavors of contract testing

1. Schema-based (provider-side)

Run the real server, hit its endpoints, and validate responses against the published schema.

┌──────────┐   real requests   ┌────────┐
│   Test   │ ────────────────> │ Server │
│  runner  │                   │        │
│          │ <──── JSON ────── │        │
└──────────┘                   └────────┘
      │
      │ validate against openapi.yaml
      ▼
      ✓ or ✗

Cheap, fast, catches 80% of drift. Tools: Dredd, Schemathesis, Postman, ShiftLeft.

2. Consumer-driven (Pact-style)

Consumers publish expectations about what they need from the producer. The producer verifies they can satisfy each consumer.

┌──────────┐                                         ┌──────────┐
│ Consumer │ ── publishes "pact" expectations ────>  │  Broker  │
│          │                                         │          │
└──────────┘                                         └─────┬────┘
                                                           │
                                                           │ consumed during CI
                                                           ▼
                                                     ┌──────────┐
                                                     │ Producer │ ── verifies
                                                     └──────────┘

Stronger, more expensive. Good for microservice fleets where many teams own different consumers. Tools: Pact, Spring Cloud Contract.

3. Snapshot-based

Record live responses; compare against the snapshot on every run. Fail if anything changed unexpectedly.

real server → capture JSON → commit to repo → compare next run

Quick to set up. Brittle (any change fails). Good for complex responses where the schema is vague.

Most mature teams combine all three: schema tests in CI for every build, Pact verification for inter-service contracts, snapshots for visual/complex responses.

What schema tests should cover

For every endpoint in the spec:

1. Every documented status code is reachable with an appropriate request.
2. Response body matches the schema for that status.
3. Required fields are always present.
4. Nullable fields are null or correct type.
5. Enums only contain values the spec declares.
6. Nested objects match their referenced schemas.
7. Response content-type matches the spec.
8. Documented headers are present (Location on 201, ETag when required, etc.).

Tools can derive all of this from an OpenAPI file. What tools can't do:

• Decide what status code a specific test should trigger (you write the request).
• Know which business-rule combinations matter.
• Generate realistic auth tokens.

Contract testing + hand-written business tests = full coverage.

Breaking vs non-breaking changes

Every contract change falls into one of three buckets:

A good contract test run classifies changes automatically. Tools: openapi-diff, graphql-inspector, pact-broker's "can-i-deploy".

The CI pipeline shape

For an API with OpenAPI:

1. Pull request opens.
2. CI runs the full schema-test suite against the branch.
3. CI computes the diff between the branch's OpenAPI file and main's.
4. Classifier marks changes as safe/dangerous/breaking.
5. Breaking changes require an explicit override label on the PR (e.g., @breaking-change-ok).
6. On merge, the OpenAPI file is published to a registry.
7. Downstream consumer repos are notified of the new version.

This turns contract drift from a gradual decay into a discrete, reviewable event.

Contract tests for async APIs

REST and GraphQL aren't the only contracts. Event-driven APIs (Kafka, RabbitMQ, SNS) also need contract testing:

• Schema registry (Confluent Schema Registry, Apicurio) stores the schema of each event.
• Producers register a schema version when publishing.
• Consumers declare the schema version they understand.
• Compatibility rules (backward, forward, full) enforce what changes are allowed.

AsyncAPI is the growing standard for documenting event-driven contracts. Tools are maturing; expect contract testing for events to be as mainstream as OpenAPI testing by 2027.

Common mistakes

1. "We'll write the OpenAPI later." Contract tests need a contract. If the spec is written after the code, the spec describes the bug, not the intent.

2. Generating the spec from code. Tempting, but means every code change is "automatically" a contract change. The contract should be the spec; the code should conform. Pick one side as authoritative.

3. Trusting the happy path. Contract tests that only cover 200 responses miss half the contract. Every declared error response needs a test too.

4. No diff-based gating in CI. If a breaking change can sneak into main, the whole apparatus fails.

5. Pact without a broker. Pact shines when consumer expectations are centralized. Without a broker, you're just running extra tests for no reason.

6. Snapshot tests that update on every PR. Developers get in the habit of snapshot --update any time a test fails. Snapshots become documentation of the latest whim, not an actual assertion.

What to automate vs what to write by hand

Automate (from OpenAPI / WSDL / SDL):

• Schema validation on every documented response.
• Negative tests per field.
• Status-code reachability.
• Breaking-change diff on every PR.

Write by hand:

• Business-rule scenarios that cross multiple endpoints.
• Flows (login → do-thing → log out).
• Real-world edge cases specific to your domain.

ShiftLeft's contract-testing workflow reads your OpenAPI, generates the automatable tests, and leaves the business scenarios for you to add on top. The split is deliberate — the stuff that can be generated shouldn't consume human time.

What's next

You've completed the Testing cluster. Next up: AI-assisted testing — how modern AI changes test generation from template-filling to genuinely useful automation.