Spot Changes in Your API Spec: An API Contract Testing Example

This is lesson 9.3 of Shift-Left API in Depth, the product track of the learning center. You need nothing installed for the concept part. For the demo, use a Professional, Enterprise or trial edition of Shift-Left API (change review isn't in the free edition) and the Store API project from lessons 9.1 and 9.2.

How do you test an API contract when the spec changes?

Here's an API contract testing example every team runs into: the API spec changes. A field becomes required. An endpoint disappears. Tests built from the old version can't tell you that on their own. So what changed, and will it break anyone?

What is API contract testing?

Your OpenAPI spec is a contract between the API and every client that calls it. Contract testing checks that both sides still keep it. That's the difference in contract testing vs API testing: an API test checks what one endpoint does, while a contract test checks that the agreement between the API and its clients still holds. And when the contract itself changes, the first question is always the same: will the clients that work today still work tomorrow?

Four changes, two of them breaking

Put version 1.1 of the store spec next to version 1.0, the one imported in lesson 9.1. Four things changed:

1. Create a new user now requires a role.
2. List users gets an optional role filter.
3. A new endpoint, Cancel an order.
4. Delete a product is gone.

To find the breaking ones, ask one question about each change: would a request that worked yesterday fail today?

• The optional filter: no. Old requests still work, so it's backward compatible.
• The new endpoint: no. Nobody calls it yet.
• The required role: yes. Every client that leaves it out now gets an error.
• The removed endpoint: yes. Every call to it fails.

Those two are breaking changes. So are a changed type, a removed enum value or a removed success response. That is how to do API contract testing when a spec moves on: compare the old version with the new one, classify each change, and update your tests. By hand, across dozens of endpoints, one missed line is enough to ship a break.

Check the Change Detection settings

Open the Store API project, built from version 1.0. Users has tests; Products has none.

1. Click Project Settings at the top right of the page, then the Change Detection tab.
2. Enable Change Detection is ticked, so every new import is compared with what's already there. Leave Auto-approve non-breaking changes unticked, so you review every change yourself.
3. If your spec lives at a URL, scroll to Monitored URLs, type the spec's address, click the plus button, then Save Monitored URLs. Click Check for Changes to compare the live spec with the project. For the sandbox spec it reports no changes.

Import the new version

This time, version 1.1 arrives as a file. Import it the way you imported the first version:

1. Click Endpoints in the sidebar, leave the API type on Auto-detect, and click Upload File.
2. Click the drop zone and pick the version 1.1 file. Auto-detect recognises it as a REST API.
3. Click Import File.

Shift-Left API doesn't overwrite your endpoints. It stops and opens Review API Changes, proposed as version 2. The change summary reads four changes in total (one new endpoint, two modified, one removed) and, in red, two breaking changes that may break existing clients.

Review every change

Go through the tabs from left to right:

• New Endpoints: Cancel an order.
• Modified: two endpoints. Expand Create a new user to see that the request body schema changed: in the old value, name and email are required; in the new value, role is required too. That's why it's marked Breaking. List users gained an optional role parameter, so it's marked Non-Breaking.
• Removed: Delete a product. Approving removes it from the project, together with its tests.
• Breaking Changes: both breaking changes in one list, Create a new user and the removed endpoint, each high severity. That's the list to share with the teams who call this API.

Approve and test the new version

Reject discards the change and nothing in the project moves. Approve & Import accepts it. The changed and new endpoints arrive as version 2, each in its own feature: Users v2 with two endpoints and Orders v2 with one. The original Users feature keeps its version 1 endpoints and their tests, and Products is down to five endpoints.

Version 2 starts with no tests. Click Users v2 in the Features table, then Create a new user, and click Generate tests. Scroll the list past the name and email tests and you'll find a missing required field test for role, rejected with a 400. That's the test that catches a client still sending yesterday's request.

Try it yourself

Download both versions of the store spec: version 1.0 and version 1.1, the lesson's copy with the four changes above. Create a project from version 1.0, then import version 1.1 and open Review API Changes. How many breaking changes does it find, and on which endpoints? Then approve, and generate tests for Cancel an order in Orders v2.

Recap

• Your spec is the contract, and an API contract testing example starts the moment it changes: ask whether yesterday's requests still work.
• A new required field or a removed endpoint breaks clients; an optional addition doesn't.
• Shift-Left API compares each new version with the last one, sorts the changes into new, modified, removed and breaking, waits for your approval, and then you generate tests for the new version.

Not every team starts from a spec. Next, in lesson 9.4, you import a Postman collection into Shift-Left API, see what each part of it becomes, and run an imported test. For more on the concept, read what API contract testing is.