Run Triage: Understand Why a Run Failed

Overview

When a run of 200 tests ends with 47 failures, the hard part is not fixing them. It is working out how many different problems those 47 failures actually are. Often 31 of them share one expired token, 11 share one changed status code, and only 5 point at something genuinely wrong with the API.

Shift-Left Studio's run triage does that sorting for you. After a run finishes, Studio groups the failures into causes and gives each cause a verdict: your tests are out of date, your test data needs attention, the environment is in the way, the API itself looks broken, or there is not enough evidence to say. Instead of 47 rows you read a headline such as "47 of 210 tests failed — 3 causes." and three cards.

Grouping and verdicts are worked out by fixed rules, with no AI and no changes to your tests. AI is used only where you allow it: to phrase the summary, to investigate causes the rules could not explain, and to propose one fix for a whole cause at once. Every AI-proposed fix waits for you to review it.

Triage runs automatically for REST, SOAP, GraphQL, JSON-RPC 2.0 / MCP and WebSocket-RPC runs, including runs executed by the local runner.

Key concepts

Before you begin

• Run some tests. Triage appears after a Test Run Pack, a workflow pack, a single run from the Run button, a bulk run of selected tests, or a local-runner upload finishes with failures. (Test Run Packs need the Professional edition or higher.)
• To use the AI parts (the plain-language summary, automatic investigation, Ask about this cause, and Fix these N tests), an administrator must configure an AI provider under AI Settings. AI calls count toward your installation's daily AI limit.
• To apply fixes you need permission to edit the tests involved.

Step 1 — Open a run's analysis

You can see triage in three places:

1. The run report. Click Reporting in the top bar, choose Functional & workflow runs, and click View Report on a run. The analysis sits above the Execution Summary. It starts folded: the headline and a one-line breakdown of verdicts (for example "● Environment 1 · ● Tests out of date 2") stay visible. Click the headline to show the cause cards. This is the fullest view and the one this article describes.
2. The Pack Execution Summary that appears when a pack finishes.
3. The Bulk Run Summary after you run a selection of tests from an endpoint. This is a compact, read-only view.

If a run was recorded before triage was available, the panel says This run has not been analysed yet. Click Work out what went wrong to analyze it. No changes are made to your tests.

Tip: Fixed a credential or a base URL and want the analysis to reflect it? Run the tests again, or click Recheck at the top right of the panel to re-read the current run.

Step 2 — Read the cause cards

Cards are ranked for you: environment problems first, then suspected product bugs, then the causes that affect the most tests. Each card shows the verdict badge, a plain title (for example "The endpoint returns 201; these tests expect 200"), its reach (for example "31 tests · 4 endpoints"), a short explanation and a hint about what to do next.

If the whole run could not reach its target, the panel shows a warning: Nothing was actually tested — the target never answered, so the rest of this run tells you nothing about your tests. Fix the environment and run again before acting on any other card.

Understanding the results

The verdicts

A small note beside the badge tells you who decided the verdict: AI second look · 85% sure when AI investigated it, or marked by you when you reclassified it. No note means a rule decided it.

What the causes look like in practice

Studio reads the response body, the headers and the request that was sent, not just the status code. That is why it can tell apart failures that look identical at first glance.

Step 3 — Open a cause and review the evidence

1. Click a cause card. Its details panel opens on the right.
2. Read the explanation, then Why we think this: the facts the verdict rests on.
3. Under What to do, read the next step. Where a single endpoint is involved, Open the first test opens it in the workbench.
4. Under What it looked like, review up to three example runs. Each shows what was Sent, what was Received, and the failed checks as Expected vs actual. Expand Request body, Request headers or Response body as needed. If the response was a web page, a note says This is a web page, not API output — the API may never have seen the call.

If no example is shown, the panel says why: none was recorded, the runs were deleted, or they could not be loaded (with Try again).

Step 4 — Ask about the cause, or get a second look

If the Project Assistant is available, the details panel includes Ask about this cause. Pick a suggested question, such as Tell me whether this is a product bug or a test problem, or tell it what the evidence cannot show, for example "that order ID does not exist on staging".

The AI reads the example requests and responses in full (credentials masked), the test's intent, how the same endpoints did elsewhere in the run, and every hint you have given. If it reads the cause differently, it offers a new verdict; you accept it, and it is recorded as your decision. Your hints also reach any fix prepared later, so "use order 1001 on staging" becomes what the fix does.

Note: Studio can also investigate unexplained causes with AI automatically after a run, within the project's AI call limit. This only explains; it never changes a test.

Step 5 — Reclassify a cause yourself

1. In the cause's details, click This is not what happened (or Say what this really is on a cause marked Needs a look).
2. Under What is this really?, optionally type what you found.
3. Choose A bug in the API, The tests are out of date, An environment problem or A test-data problem.

Your choice is shown as marked by you, and your note is kept as a hint. Marking a cause as a bug or an environment problem removes the offer to fix its tests.

Step 6 — Fix a whole cause at once

For causes marked Tests out of date, Test data or Needs a look:

1. Click Fix these N tests. Studio prepares one fix that covers the group. Fixes worked out by rules (for example, updating an expected status code) cost no AI call; otherwise one AI call covers the whole cause.
2. The Review N fixes dialog opens. Use the arrows to page through the tests. Each shows the changed fields as before (−) and after (+), with the reason.
3. Clear Include this one for any test whose change looks wrong.
4. Click Apply N fixes.
5. Read the result. The dialog lists each test with its outcome: Applied, Changed since — not applied (someone edited the test after the fix was prepared, so it was not overwritten), Test was deleted, and so on. A line such as "9 of 11 applied" tells you exactly what happened.
6. Click Re-run to check. The tests run again, and the panel shows, for example, "11 fixed · 0 still failing".

If some tests did not suit the shared fix, the dialog says how many need a look by hand.

What Studio will not do

Some fixes are refused on purpose, with the reason under Why nothing was changed.

• Suspected product bugs are never patched. A test changed to accept a server error would never report that bug again.
• Environment problems are never patched. Changing tests cannot fix a gateway, a credential or an unreachable host.
• If the target was unreachable, nothing in the run is repaired. Every other cause is a side effect of the outage.
• An expected status is never rewritten onto an error code. That would silently hide a real outage.
• A negative test is never turned into one that proves nothing. A fix may not simply drop its expected rejection; it must change the request or add a real check.
• A fix that would not change what is sent is refused, for example editing only the XML of a SOAP test whose request is built from its body parameters.
• Workflow runs are explained, not repaired. The panel says These steps belong to a workflow. Open it in the flow editor to change them.

Step 7 — Adjust the project's triage settings

1. Open Project Settings for the project.
2. Select the AI Generation tab and scroll to Failing tests.
3. Turn options on or off. They save immediately.

Troubleshooting

Best practices

• Fix Environment causes first and run again. They often explain most of the other failures.
• Treat Likely product bug as a finding to report to the API team, not a test to edit.
• Spot-check a few fixes in the review dialog, and always click Re-run to check afterwards.
• Give hints in plain words. "Customer 42 was deleted on staging" is exactly the kind of fact that changes the verdict and the fix.
• Keep Apply obvious fixes without asking off unless your team reviews changes some other way.

FAQ

Does triage change my tests? No. Grouping and verdicts never write to a test. Changes happen only when you apply fixes, or if an administrator and your project both allow rule-based fixes to be applied automatically.

How many AI calls does a run use? At most one for the summary, one per investigated cause, and one per cause you ask to fix, capped by AI calls per run. A run with 200 failures in 6 causes uses a handful of calls, not 200.

Why is a cause with a 403 marked as test data, not an access problem? Because the response body said the requested record was not found. Studio reads the body, not only the status.

Related articles

• Test Validity: What a Green Run Leaves Open
• Fix with AI: What's New in Grounded Repair
• Fix Tests with AI (existing article)
• Project Assistant: Ask Questions About Your Project
• Test run

For administrators (self-hosted installations)