Fix with AI: Grounded Repair

Overview

This article is a companion to Fix Tests with AI, which explains the basics: the Fix button, the classification, the confidence score, the before/after diff, and Apply to form, Apply & Save and Discard. Everything described there still works the same way. This article covers what has changed underneath and what you now see in the Fix with AI dialog.

In short, Fix is now grounded. Before proposing a change, Shift-Left Studio reads everything it knows about the test: the endpoint's documented contract, the requirement the test covers, what the test was meant to prove, a detailed record of how the last run actually executed, and the verdict run triage already reached about that run. Every proposed change must cite the evidence it rests on, and a change that cannot is dropped before you see it.

The result is fewer plausible-looking but wrong fixes, clear refusals where editing the test is the wrong answer, and a new Diagnostics view in the test workbench that shows how a run executed.

Key concepts

Before you begin

• Open a saved test in the test editor or workbench. Fix is offered for saved tests only.
• For the best results, run the test at least once, so Fix has a failed run to read. Without one, the dialog shows Spec-based (no run).
• Import an API specification for the endpoint, so Fix can read the documented contract.
• An AI provider must be configured under AI Settings. Some fixes are worked out without AI and cost no AI call.

Step 1 — Run Fix as before

1. Open the test and select the tab you want to repair (Request, Assertions, Response, Data or Auth).
2. Click the purple button, which is named after the tab, for example Fix Assertions. To choose a different scope, click the arrow beside it and pick Fix everything, Fix data, Fix body, Fix parameters, Fix assertions or Fix authentication.
3. The Fix with AI dialog opens and shows "Analyzing the test and proposing a fix…".

Step 2 — Read what Fix found

At the top of the dialog you see three badges:

Below the badges is a short explanation. If some evidence was too large to read in full, the dialog says so, for example "Some evidence was too large to read in full: response body." Treat that as a reason to review the change carefully.

Step 3 — Check the basis of each change

Each proposed change shows the field, its before and after values, and a basis line naming where it came from:

A change that cannot cite any of these is dropped before it reaches you. A change that would add a check on a field the evidence gives no reason to expect in the response is also dropped. A plausible but ungrounded fix is worse than none, because it gets applied.

Apply the changes with Apply to form (review in the editor, then save) or Apply & Save, or click Discard.

Step 4 — Understand refusals

When editing the test is the wrong answer, the dialog shows Why nothing was changed and offers only Close. Common reasons:

When the reason comes from the run analysis, a note adds: This is the run analysis's reading of the failure, not a fresh guess. The wording matches what the run report's cause details say, so the two screens never explain one failure two ways. If you disagree, reclassify the cause in the run report (see Run Triage) and run Fix again.

Fix also refuses, on principle:

• To turn a negative test into one that proves nothing. If the test expects the API to refuse a request, a fix may not simply switch it to expect success. It must change the request or add a real check.
• To make a change that would not change what is sent. See the SOAP note in Step 7.

Step 5 — Know when Fix needs no AI call

Some failures are resolved by a rule instead of the model. The main one is a test that contradicts itself: it accepts a set of statuses and also demands exactly one, and the API answered with a documented member of that set. Fix removes the contradictory expectation without an AI call. It does this only when the observed status is below 500, is documented in the specification's own responses, and is either a success status or the test is a negative test. Where the run has not been analyzed, the result is offered for review rather than recommended outright.

Step 6 — View a run's diagnostics

Every run now records how it executed. To view it:

1. Open the test in the workbench and select a run in the results pane.
2. Select the Request tab.
3. Expand Diagnostics.

The Diagnostics section appears only for runs that recorded diagnostics; older runs do not show it. Credentials are masked before the record leaves the runner, including runs from the local runner.

Fix reads the same record, which is why it can now tell "the token went out unresolved" apart from "the API rejected a valid token".

Step 7 — Protocol notes

Fix knows how each protocol says "no" and where an invalid value must go to reach the wire.

SOAP: body versus parameters. When a SOAP test has body parameters, those parameters are merged over the XML body when the request is built. A fix that edits only the XML would change the test but send exactly the same bytes. Fix refuses such a change and names the field that actually decides what is sent, so edit the parameter instead.

Step 8 — Convert imported Postman checks with Fix

When you import a Postman collection, most test scripts become runnable assertions. A check that uses a value the import could not follow is kept on its test, marked Needs review, with the original script as a comment. Until it is converted, it reports a failure.

To convert the checks on one test:

1. Open the test and select the Assertions tab.
2. Click Fix Assertions.
3. Fix first re-translates the check from the collection's own script, which needs no AI. If that is not enough, the AI supplies only what the check referred to (for example, that r is the response body). The check itself stays exactly as the collection wrote it.
4. Review the result in the dialog and apply it.

To convert them for a whole import, use Convert the N check(s) on the Postman import summary. Studio converts what it can from the collection's scripts without AI. Then Use the AI for the N test(s) left processes the rest 10 at a time and leaves each result on its test for you to review. Nothing converted by AI is applied without your review.

If a check no longer matters, you can delete it instead.

Troubleshooting

Best practices

• Run the test before pressing Fix, so it has real evidence to work from.
• Import or refresh the API specification. The contract is Fix's strongest evidence.
• Read the basis line before applying. A change based on "this run" alone deserves a closer look than one based on the API contract.
• When Fix refuses, treat the refusal as the answer. It usually means the problem is not in the test.
• Check Diagnostics first for failures that make no sense. An unresolved variable or a changed header explains many of them.

FAQ

Does Fix still use the last failed run? Yes, together with the specification, the requirement, the test's intent, the run diagnostics and the run analysis.

Why did Fix refuse when the model seemed confident? The run analysis's verdict outranks a fresh guess. It read the body, headers and the rest of the run, which a single test's view cannot.

Are my credentials sent to the AI provider? No. Credentials are masked in the recorded run and in the evidence Fix reads.

Is anything applied automatically? Not from the Fix button. You always choose Apply to form, Apply & Save or Discard.

Related articles

• Fix Tests with AI (existing article)
• Run Triage: Understand Why a Run Failed
• Test Validity: What a Green Run Leaves Open
• Project Assistant: Ask Questions About Your Project
• Test run

For administrators (self-hosted installations)