How to Test an OpenAPI Spec: Check the Document, Then the API

This is lesson 5.2 of OpenAPI, Contract and Schema Testing. In lesson 5.1 you learned to read a spec; now you test one, using the sandbox store's spec and, for the demo, the free edition of Shift-Left API.

How do you test an OpenAPI spec?

To test OpenAPI spec files properly, you test two things, in order. First the document: does it parse, are its rules consistent, and do its examples match its own schemas? Then the API against the document: does every operation answer with a documented status, a body that fits its schema, and a refusal for each kind of invalid input? Skip the first pass and the second inherits every mistake in the spec. Skip the second and you have a tidy document that nobody has checked against reality.

Pass one: test the document

A spec is code, so it gets the checks code gets. Three of them:

1. Validate it. A parser checks the document against the OpenAPI schema for its version and resolves every $ref. A broken reference or a wrong type for a keyword fails here.
2. Lint it. A linter such as Spectral applies style and quality rules that are valid OpenAPI but still bad practice: an operation with no 4xx response, a missing operationId, a schema with no description. You choose the rule set.
3. Check the examples. Every example should validate against the schema it illustrates.

Run the sandbox spec through those checks and two findings come up, both common in real specs:

• A 3.1 keyword form in a 3.0 document. The spec is openapi: 3.0.3, but the product price uses exclusiveMinimum: 0. In OpenAPI 3.0, exclusiveMinimum is a boolean that works with minimum; the numeric form arrived with 3.1. A strict 3.0 validator rejects it, and a lenient tool may quietly ignore the bound.
• No required fields on responses. The response schemas list properties but mark none as required, so a body with almost nothing in it still matches. As a contract, that is weak.

Neither stops a tool from importing the spec. Both change what tests can be derived from it, which is why this pass comes first.

Pass two: test the API against it

Now the running API. Take Create a new product (POST /api/v1/products). Its request schema says name is required, 1 to 100 characters; price is required and must be greater than 0; stock is an integer of at least 0; and unknown fields are not allowed (additionalProperties: false). The spec documents two answers: 201 and 400.

Here is a swagger API testing example, built straight from those rules:

Send the first three yourself:

curl -X POST 'https://demo.totalshiftleft.ai/api/v1/products' \
  -H 'Content-Type: application/json' \
  -d '{"name":"Desk Lamp","price":19.5}'

A 201. The response also shows description as an empty string, stock as 0 and category as general: the defaults the spec declares, applied by the API. That is worth a check of its own.

curl -X POST 'https://demo.totalshiftleft.ai/api/v1/products' \
  -H 'Content-Type: application/json' \
  -d '{"name":"Desk Lamp","price":0}'

A 400 with error set to VALIDATION_ERROR and a details entry naming price. Now the extra field:

curl -X POST 'https://demo.totalshiftleft.ai/api/v1/products' \
  -H 'Content-Type: application/json' \
  -d '{"name":"Desk Lamp","price":19.5,"colour":"black"}'

Also a 400, naming colour. Each refusal is documented and names the field, which is what a client needs.

The pieces of an OpenAPI testing framework

Whatever tools you pick, an OpenAPI testing framework has five parts, and each maps to one of the steps above:

Teams usually assemble this from several tools. What matters is that every part runs on every change to the spec, not only on the day it was written. For the tools commonly used at each step, read OpenAPI testing: how to test Swagger and OpenAPI APIs.

An OpenAPI response example belongs in pass one

It is tempting to assert against the examples in a spec. Don't: an OpenAPI response example is a sample for readers. In the sandbox spec, the 400 response's error field has the example BAD_REQUEST, while the API answers VALIDATION_ERROR. The schema only says error is a string, so both are valid.

Check examples in pass one, against their schemas. In pass two, assert on the schema's rules: the status, the types, the required fields, the allowed values.

Common mistakes

1. Testing only the happy path. One 201 tells you almost nothing about a schema with six rules.
2. Trusting a lenient import. A tool that accepts a slightly invalid spec is convenient, but it can drop the rule you most needed to test. Lint the spec as well.
3. Asserting the human-readable message. Assert the status, the error code and the field named in details; message wording changes.
4. Never re-running pass one. Specs change in every release. Validate and lint in the same pipeline that runs the tests.

In Shift-Left API: generate tests from the spec and run them

Shift-Left API covers the test design, the runner and the results for pass two. Its import is lenient on purpose: if strict validation fails, it still imports the dereferenced spec, so keep a linter in your pipeline for pass one.

In the free edition, with the sandbox spec imported (as in lesson 5.1):

1. Open Products, then Create a new product.
2. Click Generate tests. The coverage engine plans the tests from the spec: valid products, and negative and boundary cases for each rule in the schema, each with a severity.
3. Tick the checkbox in the table header to select every test.
4. Click Run Selected. The tests run against the environment the spec's servers entry gave the project, the live sandbox.
5. Read the Bulk Run Summary: Total, Passed, Failed and Error. When a run has failures, the summary also groups them into causes.

The OpenAPI test automation page shows where this fits with the rest of the product. Lesson 5.3 opens up the generator: which schema keywords become which tests.

Try it yourself

In the free edition, import https://demo.totalshiftleft.ai/openapi/rest.json, open Products, then Create a new product, and click Generate tests. Select every test with the header checkbox and click Run Selected. How many ran, and did any fail? Open one that failed and compare its expected status with the actual one.

Recap

• To test an OpenAPI spec, check the document first (validate, lint, check examples), then test the API against it.
• A swagger API testing example starts with one operation: each rule in its schema gives a case that must pass and a case that must be refused.
• An OpenAPI testing framework needs a parser, a linter, test design, a runner and a gate, all running on every spec change.

Next: lesson 5.3, swagger test case generation, where each schema keyword turns into test cases.