Contract-Checked API Mocks: Stop Mocks Drifting From the Spec

In this guide

1. The problem: green tests against a mock that lies
2. Five ways mocks drift
3. How to check a mock against the contract
4. Three results, not two
5. Mock the error paths, not just the happy path
6. Approaches compared
7. How Shift-Left API handles mocks
8. Mock quality checklist
9. Frequently asked questions

The problem: green tests against a mock that lies

A frontend team builds against a mock of the Orders API. Every test is green. The week of release, the real API returns total as a number and the mock returns it as a string, because someone pasted an old example into the mock a year ago. The app crashes on the first real order.

Nothing was wrong with the tests. The mock was wrong, and nothing was checking it. This is mock drift, and it is the main way mocks cost more than they save. If you are new to the topic, what API mocking is and how it differs from stubbing is a good place to start, and mocking for parallel development explains why teams use mocks at all.

Five ways mocks drift

The first three are about the contract moving. The last two are about the mock never covering it in the first place. A good approach handles both.

How to check a mock against the contract

You do not need a special tool to start. The principle is simple: take each response your mock can return, find the schema the contract documents for that operation and status code, and validate the body against it.

Here is the idea in Python, using a schema fragment from an Orders contract:

# pip install jsonschema
from jsonschema import Draft202012Validator

order_schema = {
    "type": "object",
    "required": ["id", "status", "total"],
    "properties": {
        "id": {"type": "string"},
        "status": {"enum": ["pending", "paid", "shipped"]},
        "total": {"type": "number"},
    },
}

mock_body = {"id": "ord_1001", "status": "paid", "total": "49.90"}

for error in Draft202012Validator(order_schema).iter_errors(mock_body):
    path = "/" + "/".join(str(p) for p in error.path)
    print(f"differs at {path}: {error.message}")

This prints differs at /total: '49.90' is not of type 'number', which is exactly the bug from the story above. Run it in CI against every mock and you catch drift on the day it happens.

Three practical points before you build this yourself:

• OpenAPI schemas are close to JSON Schema, but not identical. OpenAPI 3.0 uses nullable, while 3.1 aligns with JSON Schema. Our guide to OpenAPI 3.1 versus 3.0 covers the differences that affect validation.
• References need resolving. A $ref that points outside the file has to be loaded before a body can be validated against it.
• Some responses cannot be checked. More on that next, because it is the design decision that matters most.

For the wider picture of keeping specs and behaviour aligned, see API schema validation and catching drift and JSON Schema validation.

Three results, not two

A naive checker returns pass or fail. That hides the most dangerous case: a response the checker could not evaluate and quietly let through.

A trustworthy contract check has three outcomes:

Typical reasons for "not checked":

• The body is a template with placeholders, so there is no fixed value to validate.
• The body is not JSON.
• The status code is undocumented in the contract, so there is no schema to compare with.
• A $ref cannot be resolved.
• The schema uses a keyword the checker does not evaluate.

The rule is that missing evidence is never a pass. If you only ever see green, you cannot tell "verified" from "skipped". A "not checked" result with a reason tells you exactly where to look, and over time you can drive that number down.

Mock the error paths, not just the happy path

Most production incidents involve failures, not successful responses. A mock that only returns 200 teaches your client nothing about them. Four habits close the gap.

1. One named response per documented error. If the contract documents 400, 404, 409 and 500 for an operation, the mock should be able to return each, with the documented body shape.
2. Let the caller choose. A header such as X-Mock-Scenario: 404, or a query parameter, lets a test pick a response without changing the mock. Without the header, the mock answers normally.
3. Fail loudly on a typo. If the caller asks for a scenario that does not exist, return an error that lists the real names. Silently falling back to the default response turns a typo into a false pass.
4. Simulate bad networks. Add responses that hold the connection and then close it, or that drop the connection outright. These test your client's timeouts and retries, which are the code paths that hurt most in production. Always cap how long a held connection stays open.

Two more techniques are worth adding when you have the basics:

• Boundary responses. Generate variants at the schema's limits: an empty list, the longest allowed string, minimum and maximum numbers, and each optional field missing. Schemas that declare no limits give you few or none.
• Stateful behaviour. For flows such as create-then-read, let a mock remember records for a short, bounded time. Treat these as test aids with a time limit, not as a replacement for a real backend.

Approaches to keeping mocks honest, compared

These approaches are not rivals. A contract check on your mocks tells you the mock matches the documented contract. A consumer-driven contract test tells you the provider honours what a consumer needs. Our guide to API contract testing and the one on contract testing for microservices cover that second layer. Use both: the mock check keeps your test doubles honest, and the provider-side contract test keeps your spec honest.

How Shift-Left API handles contract-checked mocks

The advice above works with any stack. Here is how Shift-Left API's mock server puts it into practice. Everything in this section is available on every plan, Free included. The only plan limit is that Free caps the total number of mocks at 50.

Generated from your spec. Mocks come from an imported OpenAPI, WSDL or GraphQL definition. For REST, a generated mock gets a named response for each 4xx and 5xx the specification documents, added after the default response. They are reachable only by name, so the normal answer does not change. SOAP and GraphQL mocks get one response and no extra error variants.

Choose a response by name. Send the header X-Mock-Scenario: <name> or the query parameter ?mockScenario=<name>. An unknown name returns a 404 that lists the real names, never the default.

Fail on purpose. A response can be set to respond normally, time out (the connection is held and then closed, up to 5 minutes) or reset (the connection is destroyed). This runs before any configured delay.

The contract check. On the API's own page, "Check the mocks against the contract" compares each stored response with the schema the API documents. Every response is reported as fits, differs (with where), or not checked (with why). A template, a non-JSON body, an undocumented status, an unresolved $ref, or any keyword the checker cannot evaluate is shown as "not checked", never as a pass.

Spec changes mark mocks. When a specification change is approved, mocks of the changed API show a May be out of date chip: "The API this stands in for changed after the mock was made." You review the mock and press "I've checked it". The mark never changes how the mock answers. Our guide to OpenAPI change impact analysis covers the review that triggers it.

Keep edits when you regenerate. Regenerating a mock from the contract merges by response name: what the generator made last time, what the mock holds now, and what the generator makes today. Your edits are kept, and a preview mode shows the result first. Every change to what a mock answers is versioned and can be restored.

More controls.

• Boundary variants at the schema's limits, as an option when generating mocks.
• Stateful mocks that remember records (create, read, update, delete, list) within a project, for one hour and up to 1,000 records by default.
• A default scenario per environment, which applies only if the mock has a response with exactly that name.
• A lifecycle of draft, approved, published and retired. Only a published mock can be switched on.
• "Mock the dependencies", which creates draft, switched-off mocks for the APIs that a chosen API calls. See API dependency mapping.

What it does not do. Be clear about the limits:

• The contract check is not a full OpenAPI validator. It compares stored responses with the documented response schema, and tells you what it could not evaluate.
• A passing check does not guarantee the mock matches production. It means the mock matches the contract.
• Stateful mocks hold records for a limited time. They are not a database, so do not compare them with a real backend.
• The default scenario per environment is only a name. There is no different mock body per environment.
• Mocks are for REST, SOAP and GraphQL. Mocks for gRPC or messaging are not offered.

Setup steps are in the guides to understanding mocks and creating and configuring mocks. To compare tools on mocking features, see API mocking tools.

Mock quality checklist

• ✔ Mocks are generated from the contract, not written from memory
• ✔ Every documented error status has a named mock response
• ✔ Callers can pick a response by header or query parameter
• ✔ An unknown scenario name returns an error that lists valid names
• ✔ Timeouts and dropped connections are available for resilience tests
• ✔ A contract check runs on every mock, with three results: fits, differs, not checked
• ✔ "Not checked" results are reviewed, with a target to reduce them
• ✔ Mocks of a changed API are flagged for review after each spec change
• ✔ Regenerating a mock keeps manual edits and can be previewed
• ✔ Provider-side contract tests back up the mocks

Frequently asked questions about contract-checked mocks

What is a contract-checked mock?

A contract-checked mock is a mock whose stored responses are compared with the response schemas documented in the API contract, so you learn when the mock no longer matches the spec.

Why do mocks drift from the real API?

The spec changes and the mock does not, a developer edits a mock by hand to unblock a test, examples are copied from an old response, or the mock was generated once and never regenerated. Over time, tests pass against behaviour the real API no longer has.

Does a passing contract check prove the mock matches production?

No. It proves the stored responses fit the documented schema. If the real API deviates from its own spec, the mock inherits the spec's assumptions. Pair mocks with contract tests against the real service.

What should a contract check do with things it cannot evaluate?

Report them as not checked, with the reason, and never as a pass. A template body, a non-JSON response, an undocumented status code or an unsupported schema keyword should each be visible as a gap.

How do I test error handling against a mock?

Give the mock named responses for each documented error, and let the client pick one with a header or query parameter. An unknown name should fail loudly. Add timeouts and dropped connections to test client resilience.

Is contract-checked mocking available on every Shift-Left API plan?

Yes. The scenario, failure and contract-check features have no edition gate. The Free plan caps the total number of mocks at 50.

Is the contract check a full OpenAPI validator?

No. It compares each stored response with the schema the API documents. Anything it cannot evaluate is shown as not checked rather than passed.

Sources and further reading

• Martin Fowler: Test Double: the vocabulary for mocks, stubs and fakes.
• JSON Schema: the specification behind response validation.
• OpenAPI Specification: how responses and schemas are described.
• Pact documentation: consumer-driven contract testing against the real provider.