How to Do API Contract Testing: A Step-by-Step Method

This is lesson 5.6 of OpenAPI, Contract and Schema Testing. It pulls the earlier lessons together into one method; lesson 5.5 on schema validation is the most useful one to have read first. For the demo you need a trial edition, because Test Run Packs and CI aren't in the free edition.

How do you do API contract testing?

How to do API contract testing comes down to six steps: choose where the contract lives, make it precise, check every change to it, build a suite that checks the API conforms to it, run that suite as a gate in CI, and handle change deliberately. Most teams that "have contract tests" have done two or three of these. The value comes from doing all six, because a gap in any one lets a breaking change through.

What is API contract testing?

An API contract is the agreement between an API and its clients: which requests are accepted, and which responses come back, down to status codes, field names and types. API contract testing checks that both sides still keep it. The contract is written down, so the check doesn't depend on someone remembering how the API behaved last month.

Contract testing vs API testing vs unit testing

The comparison that confuses teams is contract testing vs API testing. A contract suite can pass while the API computes the wrong total; a functional suite can pass while a field that a mobile app reads has been renamed. Each catches what the other misses.

Step 1: choose where the contract lives

There are two main approaches:

• Spec-first (provider-driven). The OpenAPI spec is the contract. The provider tests that it conforms; clients generate code and mocks from the spec. This suits public APIs and APIs with many clients you don't control.
• Consumer-driven. Each client writes down the requests it makes and the fields it reads, and the provider verifies every client's contract. Pact is the best-known tool; lesson 5.7 covers it.

Many organisations use spec-first for their public surface and consumer-driven contracts between internal services.

Step 2: make the contract precise

A contract can only enforce what it states. Before writing tests, tighten the spec:

• Mark response fields as required where clients depend on them.
• Use enum for fixed sets: statuses, roles, error codes.
• Document every status the API really returns, including its 400s.
• Define one error envelope and use it everywhere.

The sandbox spec is a useful warning: its response schemas mark nothing as required, so a contract test can check types but not presence.

Step 3: check every change to the contract

Every pull request that touches the spec should be diffed against the last released version and each change classified as breaking or not. A new required request field, a removed endpoint, a narrowed enum or a changed type is breaking; a new optional field or endpoint isn't. This catches a break before any code runs.

Step 4: build a conformance suite

The provider-side suite checks that the running API matches the contract, operation by operation:

1. Each documented success status, with a body that fits the response schema.
2. Each documented error status, with the error envelope.
3. Each request rule enforced: missing required fields, wrong types, values past their limits.
4. Nothing undocumented: no status, and no field type, that the spec doesn't describe.

What the suite deliberately leaves out: business logic, performance and data correctness. Those belong to functional and performance suites.

Step 5: gate CI on it

Run the contract suite on every pull request to the provider and before every deploy. Fail the build below a pass rate you choose, and publish the results where both the provider and client teams can see them. A contract suite nobody runs is documentation.

Step 6: handle change deliberately

When the contract has to break, don't break it in place. Add a new version, keep the old one working for a stated period, tell clients, and keep testing both until the old one is retired. Lesson 5.8 covers versioning.

Common mistakes

1. Treating the spec as documentation only. If nothing fails when the API and spec disagree, the spec will drift.
2. Putting business rules in the contract suite. It becomes slow and flaky, and teams stop trusting it.
3. Running it only on the provider. Clients should see the result, or at least be told about breaking changes.
4. "Fixing" a failing contract test by editing it. First decide whether the API or the contract is wrong.

For the three common approaches and their tools, read API contract testing: what it is, how it works and tools.

In Shift-Left API: a contract suite as a Test Run Pack

In Shift-Left API the spec is the contract. Steps 3 to 5 map onto three features: change detection on re-import (step 3, walked through in lesson 9.3), tests generated from the spec (step 4), and Test Run Packs that a pipeline runs (step 5). Test Run Packs, schedules and CI come with Professional, Enterprise and the 15-day trial.

1. Import https://demo.totalshiftleft.ai/openapi/rest.json and open Products. Click Generate Tests for All: every Products endpoint gets tests for its documented statuses, its request rules and its response outline.
2. Click Execution in the top bar, then New Test Run Pack. Choose Functional, and name it Store API contract.
3. On step two, pick the project and tick the Products tests. Finish the wizard; lesson 9.10 walks through all five steps.
4. Click Run Now. The summary shows passed, failed and total. View results opens the run in Reporting, with the execution type Manual.
5. To make it a gate, run the same pack from your pipeline with a minimum pass rate, using the ready-made GitHub Action, Azure DevOps task or Jenkins plugin. Lesson 9.15 shows the pipeline step and the pass-rate rule.

Approved spec changes create new endpoint versions; existing tests are never rewritten to match. When a contract test fails, you decide whether the API or the spec is wrong. The API contract testing page shows the rest of the workflow.

Try it yourself

With a trial, import the sandbox spec, generate tests for every Products endpoint, put them in a Test Run Pack in Execution, and click Run Now. Read the summary, then find the run in Reporting: what is its execution type?

Recap

• How to do API contract testing: choose the contract, make it precise, diff every change, build a conformance suite, gate CI on it, and version breaking changes.
• What is API contract testing? Checking that an API and its clients still agree on the interface, against a written contract.
• Contract testing vs API testing: one checks the interface, the other the behaviour, and contract testing vs unit testing is the boundary versus one piece of code. You need all three.

Next: lesson 5.7, what Pact contract testing is, the consumer-driven side of step 1.