What Is Swagger Testing? Reading an OpenAPI Spec Like a Tester

This is lesson 5.1 of OpenAPI, Contract and Schema Testing. It is the first lesson of the track and needs no setup: if you have sent a request before, as in request and response anatomy, you know enough.

What is swagger testing?

Swagger testing means testing an API against its Swagger or OpenAPI document. The document says what each endpoint accepts and what it returns; the tests check that the running API keeps that promise. The spec is your expected result, written down before anyone sends a request. The API's real answers are the actual result. Every difference between the two is either a bug in the API or a bug in the spec, and both are worth finding.

So before you can test against a spec, you need to read one. This lesson shows you what to look for, using the sandbox store's spec at https://demo.totalshiftleft.ai/openapi/rest.json.

Swagger or OpenAPI: one format, two names

Swagger was the original name of the format. In 2015 the Swagger specification was donated to the newly formed OpenAPI Initiative, and in 2016 it was renamed the OpenAPI Specification; version 3.0 followed in 2017 and 3.1 in 2021. "Swagger" now mostly names a family of tools around the format, such as Swagger UI and Swagger Editor. People still say "the Swagger file" for both, so in practice:

• A document with "swagger": "2.0" at the top is a Swagger 2.0 spec.
• A document with "openapi": "3.0.3" or "3.1.0" is an OpenAPI 3 spec. The sandbox's spec is 3.0.3.

The testing ideas in this track work the same for both.

The parts of a spec a tester reads

A spec is long, but a tester needs only a few parts of it:

Each limit in that table is a test waiting to be written. limit has a maximum of 100, so a request for 101 should be refused. name is required, so a product without one should be refused. You'll turn those limits into tests in lesson 5.3.

Reading OpenAPI spec responses

The responses block is the part testers read most. For Get a single product by ID, the sandbox spec documents two answers:

• 200: a body with success (a boolean) and data, a product with id, name, price, description, stock, category, created_at and updated_at.
• 404: "Record not found", with success, error, message and a few more fields.

Send both requests and compare them with what the spec promised:

curl -X GET 'https://demo.totalshiftleft.ai/api/v1/products/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa'

The Wireless Mouse comes back with a 200, a name that is a string and a price that is a number, as documented. Now ask for a product that doesn't exist:

curl -X GET 'https://demo.totalshiftleft.ai/api/v1/products/00000000-0000-0000-0000-000000000000'

A 404, with error set to NOT_FOUND. Both answers are on the documented list, so both pass this first check.

Look again at the 200 body. The spec lists updated_at, but the seeded product doesn't send it. Is that a bug? Here the spec's response schema marks no field as required, so a missing updated_at is allowed. That is worth noticing: a response schema with no required list makes a weak contract, because a body with almost nothing in it would still match.

A swagger response example is not the contract

Many specs carry examples, and they help a reader. In the sandbox spec, the error field of the 404 response has the example BAD_REQUEST, but the API you just called answered NOT_FOUND. Nothing is broken: the schema says error is a string, and NOT_FOUND is a string.

That is the difference between the two:

• A schema is a rule every body must follow: types, required fields, allowed values.
• An example is one sample body that someone typed in to illustrate the response.

Examples drift out of date without breaking anything, so a swagger response example is documentation, not something to assert against. If a value really is fixed, such as an error code, the spec should say so with an enum in the schema. Then a test can check it.

What swagger testing an API checks

With the spec read, swagger testing an API comes down to five questions you ask of every operation:

1. Does every documented operation answer? A path in the spec that returns 404 for every request is either unbuilt or misdocumented.
2. Are the status codes on the documented list? A 200 where the spec says 201, or a 400 where it documents only 200 and 404, is a difference to resolve.
3. Does each body match its schema? Types, required fields and allowed values.
4. Is invalid input refused? Each limit in the parameters and request body should be enforced, with a 4xx and a clear error.
5. Is the spec itself sound? A spec that doesn't parse, or whose rules contradict each other, produces wrong tests. Lesson 5.2 covers testing the document as well as the API.

For the wider picture, including the tools teams use for each question, read OpenAPI testing: how to test Swagger and OpenAPI APIs.

In Shift-Left API: import a spec and read what it creates

Shift-Left API reads the same parts of the spec you just read, and turns them into a project you can test. This works in the free edition. (Lesson 9.1 walks through the project screens in more detail.)

1. Click Project, name it, and create it. A new project opens straight into Import API.
2. Leave the API type on Auto-detect, paste https://demo.totalshiftleft.ai/openapi/rest.json, and import it.
3. Read the result. The spec's tags become features: Users, Products, Orders, Demo Data and Protected Store, with 38 endpoints between them. The Swagger view says there are no tests yet: a spec describes endpoints, and the tests come next.
4. Open Products. It holds six endpoints, one for each operation the spec tags as Products.
5. Double-click Get a single product by ID, or click Edit on its row. Every field came from the spec: the method, path, summary and operation ID, the id path parameter, and the responses, 200 and 404.

The same import reads Swagger 2.0 and OpenAPI 3.x in JSON or YAML, by URL, file upload or paste. The OpenAPI test automation page lists the other formats.

Try it yourself

In the free edition, create a project and import the sandbox spec from https://demo.totalshiftleft.ai/openapi/rest.json. Open Products: how many endpoints does it hold? Open Get a single product by ID: which responses does it document, and which fields does the 200 body list?

Recap

• What is swagger testing? Checking a running API against its Swagger or OpenAPI document: the spec is the expected result.
• OpenAPI spec responses list each status code an operation may return and the schema of its body; a response schema with no required fields is a weak contract.
• A swagger response example illustrates a body; the schema is the rule, so assert against the schema.

Next: lesson 5.2, how to test an OpenAPI spec, where you check the document itself, then generate tests from it and run them.