What Is Schema Validation in API Testing? Requests, Responses and Drift

This is lesson 5.5 of OpenAPI, Contract and Schema Testing. Lesson 5.4 validated one response against one schema; this lesson zooms out to the whole API, and to the spec itself as it changes.

What is schema validation in API testing?

Schema validation in API testing checks API traffic, and the API's own definition, against its schema. It happens in three places: on requests, so invalid input is refused; on responses, so clients get the shape they were promised; and on the spec, so each new version is compared with the last before it ships. When the schema and the running API disagree, you have drift, and drift is what schema validation exists to find.

What is an API schema?

An API schema is the machine-readable definition of an API: its operations, their inputs and outputs, and the shapes of the data. For REST that is usually an OpenAPI document, whose data shapes are written in JSON Schema. GraphQL has its SDL; SOAP has a WSDL with XSD types. Everything in this track reads the schema: tests, mocks, documentation and generated client code. That is why a wrong schema costs more than a wrong comment.

Three places to validate

Request validation is the API's job, but testing it is yours: every rule in the request schema should produce a refusal, as you saw in lesson 5.2. Response validation and spec comparison are where tests earn their keep.

An API schema validation example: drift in the sandbox

The sandbox store is a well-behaved API, and it still has drift. First, a request the spec covers:

curl -X POST 'https://demo.totalshiftleft.ai/api/v1/products' \
  -H 'Content-Type: application/json' \
  -d '{"name":"Desk Lamp","price":0}'

A 400 with a details entry naming price. The spec documents a 400 for this operation, so request and response both match the schema.

Now a malformed ID on Get a single product by ID:

curl -X GET 'https://demo.totalshiftleft.ai/api/v1/products/not-a-uuid'

A 400, with error set to VALIDATION_ERROR. But for this endpoint the spec documents only 200 and 404. The API is doing something sensible, refusing an ID that isn't a UUID, and the spec doesn't say so. A client generated from the spec has no code path for that 400.

Two more pieces of drift you've already met in this track:

• A documented field that never arrives. The product schema lists updated_at; the seeded products don't send it. Because the response schema marks nothing as required, no validator can flag it.
• An example that disagrees with reality. The documented 404 example says BAD_REQUEST; the API says NOT_FOUND. The schema only says "a string", so that passes too.

Each one is either an API bug or a spec bug, and only a person can say which. Schema validation's job is to make sure someone is asked.

Kinds of drift, and which check finds them

Read API schema validation: what it is, how to do it, tools for where gateways and CI fit in.

Common mistakes

1. Validating only status codes. A 200 with the wrong shape is still a broken response.
2. Weak response schemas. Without required and enum, response validation can only check types.
3. Fixing drift by editing the test. If a test starts failing because the response changed, decide whether the API or the spec is wrong first. Changing the test to match makes the change permanent and invisible.
4. Comparing specs by eye. Diffs across dozens of endpoints need a tool and a rule for what counts as breaking.

In Shift-Left API: both kinds of check

Shift-Left API checks drift in both directions.

Responses against the spec. After each run, the responses of tests that passed are compared with the documented response schema: required fields present, primitive types and enum values. A passing test that got a status the spec doesn't list is reported as Undocumented status; a body that doesn't fit is reported as No longer matches the contract. These findings never change a run's result, and nothing is changed automatically. This is part of failure analysis, in every edition.

The spec against its last version. Re-importing a spec compares it with the project's current endpoints and classifies each change as breaking or non-breaking. Examples of breaking: a new required parameter, a removed success response, a narrowed enum, a type change. To set it up:

1. Click Project Settings, then the Change Detection tab.
2. Check that Enable Change Detection is ticked. Leave Auto-approve non-breaking changes unticked so you review every change.
3. Under Monitored URLs, type https://demo.totalshiftleft.ai/openapi/rest.json, click the plus button, then Save Monitored URLs.
4. Click Check for Changes. The live sandbox spec matches the project, so it reports no changes.

When a spec does change, the import stops at Review API Changes, with new, modified, removed and breaking changes on separate tabs, until you approve or reject the whole set. Detection works in every edition; approving a change set needs Professional, Enterprise or the trial. Lesson 9.3 walks through a full review with two versions of the store spec, and the API change detection page has the rules.

Try it yourself

In a project built from the sandbox spec, open Get a single product by ID and note the responses it documents. Then add a test on it with the id path parameter set to not-a-uuid, run it, and open the Response tab. Which status came back, and is it on the documented list?

Recap

• What is schema validation in API testing? Checking requests, responses and the spec itself against the API schema, so drift between them is found.
• What is API schema validation worth on a weak schema? Little more than type checks: add required and enum so missing fields and new values can't pass unnoticed.
• This API schema validation example, a 400 the spec never documents, shows drift isn't always a bug in the API; sometimes the spec is behind.

Next: lesson 5.6, how to do API contract testing, which turns these checks into a suite that gates every release.