What Should an API Test Coverage Tool Measure?

This is lesson 5.9 of OpenAPI, Contract and Schema Testing, the last in the track. It uses ideas from lesson 5.3, where each schema rule became a test case; here you measure how many of those cases a suite really has.

What should an API test coverage tool measure?

An API test coverage tool should measure your tests against what the API documents, in several dimensions, not just one. "Every endpoint has a test" is the most common coverage claim and the least useful: one happy-path test per operation reads 100% endpoint coverage while leaving every error status, every parameter and every request rule untested. A useful tool counts status codes, parameters, body fields and assertions as well, each against the spec, and shows you the gaps by name.

The dimensions of REST API test coverage

Two principles make the numbers honest. First, a test only counts for a dimension when it asserts something: calling an endpoint and ignoring the answer covers nothing. Second, a dimension the spec doesn't describe is not applicable, not 100%: an operation with no request body has no body-field coverage to report.

OpenAPI test coverage, worked by hand

Take List products (GET /api/v1/products) from the sandbox spec. It documents one response, 200, and declares seven query parameters: page, limit, sort and order, plus three simulation parameters, delay, error and random_fail. It has no request body and no security scheme.

Now suppose the suite has one test: it calls List products with no parameters and asserts a 200.

By endpoint and status code, this endpoint is "fully covered". By parameters it is at zero, and nothing checks that limit above 100 is refused, even though the spec says it must be. That is why OpenAPI test coverage has to be multi-dimensional.

Notice the simulation parameters too. A coverage tool counts what the spec declares, so delay, error and random_fail count against you until a test sends them. Either test them or accept the gap knowingly; don't let it hide in an average.

How to measure API test coverage, step by step

1. Take denominators from the spec, per operation: responses, parameters, body fields, security.
2. Take numerators from the tests: what each one sends, and what it asserts.
3. Exclude what doesn't apply instead of counting it as covered.
4. Report per operation and per dimension, not one project-wide number. An average hides a critical endpoint at zero.
5. List the gaps by name ("no test sends limit", "no test asserts the 404"), so each one is a task.
6. Recompute on every spec change. A new parameter is a new gap the day it ships.

Coverage measured this way is spec coverage. It is different from code coverage, which counts lines of the server executed, and from traffic coverage, which compares tests with real production calls. Each answers a different question; spec coverage is the one a contract owner needs. For report contents and tool criteria, read API test coverage: how to measure it, dimensions and tools.

Common coverage traps

1. Counting calls, not checks. A suite of status is not 500 tests reaches high coverage and catches little.
2. One project-wide percentage. It rewards many easy endpoints over a few hard ones.
3. Ignoring negative scenarios. Happy paths alone can fill most dimensions in the table except the one that finds bugs.
4. Counting stale tests. Tests for an endpoint version that no longer exists shouldn't count.

In Shift-Left API: the endpoint coverage panel

Shift-Left API measures coverage against the spec and against the scenarios its generator plans for each endpoint, not against production traffic or server code. Coverage on an endpoint is available in every edition; the project-wide coverage figure on Analytics → Project Analytics needs Professional, Enterprise or the trial. The API test coverage page has the full picture.

1. With the sandbox spec imported, open Products, then List products, and click Generate tests.
2. On the endpoint page, click Endpoint functional coverage to open the panel.
3. Read the first line. Overall is functional coverage: scenarios, status codes, parameters and request body. Production-grade adds seven checks: schema validation, Content-Type, auth depth (401 and 403), method routing, a performance assertion, data integrity and error-body validation. Because coverage is measured against the scenarios the generator plans, one test can't make an endpoint read 100%.
4. Read the counts: Scenarios, Status codes (out of 1), Parameters (out of 7) and Request body.
5. Read the Coverage verdict: one line per dimension, with auth depth marked not applicable, because List products has no security scheme.
6. Read the gap list. Each gap is one of 11 types, explained in plain language with the test that would close it.
7. Click Generate tests for gaps. If a gap stays open because your project's own settings rule it out, such as performance switched off, it is flagged rather than silently skipped.

Tests marked stale after a regeneration never count as coverage.

Try it yourself

Import the sandbox spec, open Products, then List products, and click Generate tests. Open Endpoint functional coverage. What are the Overall and Production-grade scores? How many of the 7 parameters are covered, and which verdict line is marked not applicable?

Recap

• An API test coverage tool should measure against the spec, in several dimensions, and name each gap.
• REST API test coverage counts status codes, parameters, body fields and assertions, not just endpoints; a dimension the spec doesn't describe is not applicable.
• How to measure API test coverage honestly: denominators from the spec, numerators from what tests send and assert, reported per operation.

That completes the track. Next: Test Data, Mocks and End-to-End API Testing, starting with what a mock API is.