JSON Schema Validation Example: One Schema, Three Payloads

This is lesson 5.4 of OpenAPI, Contract and Schema Testing. In lesson 5.3 the schema's rules built requests; here they check responses. You'll need nothing installed for the concept part, and a trial or the free edition of Shift-Left API for the demo.

What is JSON Schema validation?

JSON Schema validation checks a JSON document against a schema that says what shape the document must have. This lesson is a JSON Schema validation example from start to finish: one schema for a real API response, three payloads run through it, and the error each one produces. The schema lists the required fields, the type of each one and the values it may take; the validator either accepts the document or lists every rule it breaks, with where and why.

The payload

Here is the response the sandbox store sends for its product list, two items per page:

curl -X GET 'https://demo.totalshiftleft.ai/api/v1/products?limit=2'

It has three parts: success, a data array of products, and a pagination object with total, page, limit, pages, hasNext and hasPrev.

The schema

The sandbox's own spec describes this response but marks no field as required, so almost any object would pass it. For a useful contract, here is a stricter schema for the same body:

{
  "type": "object",
  "required": ["success", "data", "pagination"],
  "properties": {
    "success": { "type": "boolean" },
    "data": {
      "type": "array",
      "maxItems": 100,
      "items": {
        "type": "object",
        "required": ["id", "name", "price", "stock"],
        "properties": {
          "id":    { "type": "string", "format": "uuid" },
          "name":  { "type": "string", "minLength": 1, "maxLength": 100 },
          "price": { "type": "number", "exclusiveMinimum": 0 },
          "stock": { "type": "integer", "minimum": 0 }
        }
      }
    },
    "pagination": {
      "type": "object",
      "required": ["total", "page", "limit"],
      "properties": {
        "total": { "type": "integer", "minimum": 0 },
        "page":  { "type": "integer", "minimum": 1 },
        "limit": { "type": "integer", "minimum": 1, "maximum": 100 }
      }
    }
  }
}

The limits come from the request side of the same spec: a product's name is 1 to 100 characters, its price is above 0, its stock is at least 0, and a page holds at most 100 items.

Three payloads through one schema

Read payload 3 closely. The validator reports both problems, not just the first, and each error points at a location in the document (a JSON Pointer such as /data/1/stock) and at the keyword that failed. That is what makes schema errors fast to act on. Payload 2 is the classic drift: a backend change starts serialising a decimal as a string, the status is still 200, and every client that does arithmetic on price breaks.

JSON Schema validation rules that matter for APIs

One dialect point: OpenAPI 3.1 uses JSON Schema's 2020-12 draft, while OpenAPI 3.0 uses an adapted subset of an older draft. In 3.0, nullable: true stands in for a null type and exclusiveMinimum is a boolean. Validate a spec's schemas with a validator set to the dialect the spec declares. For code in Python, JavaScript and Java, read JSON Schema validation: rules, examples and API testing.

JSON Schema validation in API testing

In a test suite, schema validation sits next to value checks, not instead of them:

• Shape checks catch contract breakage: a missing field, a changed type, a new enum value. They pass on any well-formed body.
• Value checks catch wrong data: a price of 0.01 instead of 24.99 is the right type and the wrong answer.

The usual pattern is one schema check per response in every positive test, plus a few value checks on the fields the test is about. Run them on every build, so a change to the response shape fails before it merges. Lesson 5.5 widens this to schema validation of the whole API, including drift between the spec and what the API really does.

In Shift-Left API: the shape checks it really runs

Be clear on one point first: Shift-Left API has no JSON Schema assertion type. Nothing in its assertion catalogue validates a whole body against a JSON Schema or against the spec's response schema in one step. What it does have is four things, each narrower:

1. Generated tests check the outline. Most tests generated from the spec carry a baseline of the status code, a JSON body and the response time; tests aimed at the schema and contract dimension also assert that the body is an object.
2. Quick JSON assertions check fields by path. In the test workbench, a Quick JSON assertion picks a field from the endpoint's response schema (or you type the path) and applies an operator: exists, does not exist, is of type, is one of, array length at most, and more. The types are JSON's: string, number, boolean, array, object and null. There is no integer, so stock is checked as a number.
3. The coverage panel counts structure checks. An endpoint's Schema / contract verdict turns green once any of its tests asserts the body's type (JSON, object or array) or uses a field exists, field type or field equals assertion.
4. Failure analysis compares passing responses with the spec. After a run, the responses of tests that passed are compared with the documented response schema: required fields present, primitive types and enum values. A mismatch is listed as "No longer matches the contract". It never changes the run's result and is never fixed automatically, because the drift may be the bug. Fields typed with oneOf, or not typed at all, are not checked.

One limit to know: a path into a list, such as data.price, reads the first item that has the field, and data[0].price reads the first item. Either way the check covers one item, not every item, where JSON Schema's items covers them all. For a rule across every element, write a custom assertion in JavaScript, for example return response.data.data.every(p => typeof p.price === 'number');.

Add shape checks to a List products test

With the sandbox spec imported and a test on List products (generated, or added with Add Test):

1. Open the test in the test workbench and click the Assertions tab.
2. Click Add Assertion, open Type, and choose Quick JSON assertion under Fields. Set the field to data, the operator to is of type and the type to array.
3. Add a second: data, array length at most, 10, the sandbox's default page size.
4. Add a third: type the path data[0].price, is of type, number.
5. Add a fourth: pagination.total, is of type, number.
6. Click Run. Each check shows Expected and Actual.

Now break one on purpose: change the third to expect string and run again. It fails, with Expected reading type string and Actual showing the price, a number. That is payload 2 from the table, caught by a test. The assertion types page lists the full catalogue.

Try it yourself

On a test for List products, add Quick JSON assertions: data is of type array, data[0].price is of type number and pagination.total is of type number. Run it. Then change the expected type of data[0].price to string, run it again, and read the failing check's Expected and Actual.

Recap

• What is JSON Schema validation? Checking a JSON body against a schema of required fields, types and allowed values, with every broken rule reported by location and keyword.
• This JSON Schema validation example showed type, required and items catching the breakage that matters most; format and additionalProperties need care.
• For JSON Schema validation in API testing, pair shape checks with value checks; in Shift-Left API, shape checks are field-level assertions plus a contract comparison on passing runs, not a JSON Schema assertion.

Next: lesson 5.5, what schema validation in API testing covers, from single responses to drift between the spec and the API.