Test Validity: What a Green Run Leaves Open

Overview

A green run is good news, but it does not always mean what it seems to. A test that only checks "the status was 200" stays green even if the endpoint starts returning an empty body. A requirement with three passing tests is not verified if none of those tests checks what the requirement says. And a response can quietly stop matching its API documentation while every test still passes.

After every run, Shift-Left Studio also reads the tests that passed and reports what the green result leaves open. You see these findings under Passed, but worth a look, below the failure causes (or on their own when everything passed). Studio also rolls the run up into a verdict per requirement, so you can tell "all its tests passed" apart from "something actually checked it".

These checks use no AI and never change a test by themselves. Where a safe change exists, Studio prepares it for you to review: missing checks to add to a passing test, or a better request for a negative test that was refused for the wrong reason. Nothing is ever applied automatically.

Key concepts

Before you begin

• Run your tests. Findings are produced for single runs, bulk runs, Test Run Packs and workflow packs. (Test Run Packs need the Professional edition or higher.)
• For contract comparison, the endpoints need a response schema, which comes from an imported API specification.
• For requirement verdicts, the project needs requirements linked to tests, typically through requirement-based test generation.
• To apply suggestions, you need permission to edit the tests.

Step 1 — Find the passing-test findings

1. Click Reporting in the top bar, choose Functional & workflow runs, and click View Report on a run.
2. In the run analysis above the Execution Summary, expand the panel by clicking its headline if it is folded.
3. Scroll below the failure causes to Passed, but worth a look. If everything passed, the panel says Everything passed. and the findings follow it. A folded panel also shows a count such as "3 passing checks worth a look".

If there are no findings but the run touched requirements, the section is titled What this run verified and shows only the requirement summary.

Understanding the results

Finding types

Requirement verdicts

Important: "All its tests passed" is not the same as Verified. A requirement is verified only when a check derived from it actually ran and passed. A green test that asserts only statusCodeEquals 200 proves nothing about a requirement that says "the discount must not exceed 20%".

The run analysis shows a one-line summary such as "Requirements: 1 failing · 2 not really verified · 3 unverified · 41 verified." with the requirements that need action listed underneath.

Contract comparison

For REST, GraphQL and JSON-RPC responses, Studio compares the response it observed with the response schema in your specification: required fields present, basic types correct, and values inside documented lists. Fields without a clear type are skipped rather than guessed.

Contract comparison is not performed for SOAP (XML schema checking is not part of this check) or WebSocket-RPC (a WebSocket call has no HTTP response to compare). Findings of this kind never appear for those protocols.

A contract mismatch never changes a run's status. Specifications go out of date, and a strict check would turn green suites red on the first day.

Step 2 — Open a finding

1. Click a finding card. Its details open on the right.
2. Read the explanation, then What we saw: the evidence, such as which fields were checked, which documented field was missing, or which status came back.
3. Under Where, see the endpoints involved.
4. Under Tests, click Open test next to a test to open it in the workbench. (This list appears when the finding covers a single endpoint.)

Step 3 — Add missing checks to passing tests

For Passed without really checking and Requirement not really verified:

1. In the finding's details, click Add checks to these N tests (or Add checks to this test).
2. Wait while Studio works out what to add. The checks come from your API documentation and from the requirement, not from AI.
3. When you see "Prepared N changes for review", click Review N changes.
4. In the review dialog, page through the tests, clear Include this one for any you do not want, and click Apply N fixes.
5. Re-run the tests to confirm the new checks pass. The details panel reminds you: "N tests updated. Re-run them to check the new checks pass."

Two rules keep this safe:

• It only ever adds. A suggestion never removes a check, weakens an existing one, or changes the request. A suggestion that tries is refused.
• It never asserts a field the response did not contain. Otherwise a green test would turn red on its next run, and the feature meant to help would be what broke your suite.

Step 4 — Fix the request of a negative test

For Rejected for the wrong reason:

1. Click Fix the request in these N tests.
2. Studio rebuilds each test's request so it sends a value its own specification forbids, and the API must refuse it for the reason the test names. Every check the test already makes is kept exactly as it is.
3. Review and apply the changes as in Step 3, then re-run.

Where a rebuild is not possible, the test is left out with a reason. Common reasons:

• SOAP: the element is a plain string (xsd:string). XML carries no type, so every value is well-formed and nothing can provoke a type error.
• WebSocket-RPC: the method changes data on the server, so Studio will not send a crafted call to it.
• The rebuilt request would still match a request that a passing positive test already sends, so it would not prove anything new.

For SOAP tests, the invalid value is placed in the test's body parameters rather than the raw XML, because that is what decides what is actually sent.

Step 5 — Handle findings that are never fixed automatically

No longer matches the contract and Undocumented status have no fix button. Their details explain under Why nothing was changed: rewriting the tests to accept a changed response would hide a possible defect in the API. Either the API needs fixing, or its specification is out of date, and only you can say which.

To act on one:

1. Check the finding's evidence against the API team's intended behavior.
2. If the API is wrong, raise it with the API team.
3. If the specification is out of date, update it and re-import it, so future tests and comparisons use the correct contract.

Requirement unverified also has no fix button, because there is nothing to change. Generate tests for the requirement, link it to an endpoint, or include its tests in the next run.

Step 6 — See requirement verdicts across runs

The run analysis shows the verdicts for one run. To see the latest verdict for every requirement in the project:

1. In the run analysis, click See all next to Requirements. The project's Traceability tab opens.
2. In the Requirements Traceability Matrix, read the Verified column. It shows verified, not really, failing, not run or unverified, using the most recent real verdict from recent runs. Hover over a value to see its evidence.

The requirements review list also shows a verdict chip on each requirement, such as verified by tests or not really verified.

Step 7 — Ask about a finding

If the Project Assistant is available, each finding's details include Ask about this. Suggested questions include Show what this test is not checking and Explain whether it would catch a regression. Where no fix is offered, you can also ask Explain why this cannot be fixed automatically.

Step 8 — Turn the checks on or off for a project

1. Open Project Settings, select the AI Generation tab, and scroll to Failing tests.
2. Under Understanding failures, use these options:

There is deliberately no option to apply these suggestions automatically. A passing test gives no sign that anyone wanted it changed.

Troubleshooting

Best practices

• Read Passed, but worth a look on green runs too; that is where it matters most.
• Treat contract findings as questions for the API team, not test problems.
• Use Verified in the traceability matrix, not the test count, when reporting requirement coverage.
• Re-run tests right after applying suggestions to confirm the new checks hold.

FAQ

Will these findings make my run fail? No. They never change a run's status.

Does Studio use AI for these checks? No. The findings, the verdicts and the suggested checks are all worked out from your tests, your specification and your requirements.

Can a suggestion remove one of my assertions? No. Add checks only adds. Fix the request changes the request and keeps every assertion unchanged.

Why is a requirement "unverified" if it has tests? Its tests may not have run in this run, or it may have no machine-checkable outcome. The evidence line says which.

Related articles

• Run Triage: Understand Why a Run Failed
• Fix with AI: What's New in Grounded Repair
• Project Assistant: Ask Questions About Your Project
• Test run

For administrators (self-hosted installations)