Azure DevOps API Test Cases: From Test Plans to a Pipeline Gate

This is lesson 7.4 of API Testing in CI/CD, Regression and Performance. Lesson 7.3 ran a test pack from Jenkins. Azure DevOps adds something Jenkins doesn't have: a place to write and track test cases next to the pipeline that runs them.

What are Azure DevOps API test cases?

Azure DevOps API test cases are the checks for your API, held in Azure DevOps in one of two forms. In Azure Test Plans, a test case is a work item: a title, steps and expected results that a person writes, reviews and runs or tracks. In Azure Pipelines, an API test case is an automated test that a pipeline step runs on every build, with each result recorded on the run's Tests tab. The useful setup connects the two: the case is described once, and automation proves it on every change.

Test Plans and Pipelines: two homes for one test case

The common mistake is to keep both copies by hand. The case in Test Plans says "returns 404 for an unknown product", the automated test checks something slightly different, and nobody notices for a year. Decide which one is the source of truth: for API tests it is usually the automated test, with the Test Plans case kept as the readable description and the link to the requirement.

A worked API test case, written for Test Plans

Here is one case for the sandbox store at https://demo.totalshiftleft.ai, written the way a Test Case work item holds it:

Three things make an API test case useful rather than decorative: one behaviour per case, the exact request (method, path, the IDs that matter), and an expected result someone can check without guessing. "Product endpoint works" is not a test case; the table above is.

Azure DevOps REST API test cases: scripting the plan

Teams with many APIs don't type test cases into the web UI. Azure DevOps exposes REST APIs for all of it, authenticated with a personal access token (PAT):

• Create or update a test case. A Test Case is a work item, so the Work Item Tracking API creates it with a JSON Patch document: POST https://dev.azure.com/{organization}/{project}/_apis/wit/workitems/$Test%20Case. The steps go in the Microsoft.VSTS.TCM.Steps field.
• Put it in a plan. The Test Plans API manages plans, suites and the cases in them, under _apis/testplan/Plans/{planId}/Suites/{suiteId}/TestCase.
• Record a run. The Test API creates test runs and adds results to them, under _apis/test/runs.

That last one is what pipeline tasks do for you. PublishTestResults@2 reads a JUnit file and creates the run and its results, so a pipeline doesn't need its own script to record results. Use the REST API when you want something the task doesn't do, such as generating cases from your OpenAPI spec into a suite.

Tests also flow the other way. Shift-Left API reads an Azure DevOps CSV export of work items as requirements input, keeping each item's acceptance criteria, so the backlog can drive test generation. That extraction needs an AI provider to be configured.

Running API test cases in Azure Pipelines

In a YAML pipeline, API tests usually sit in their own stage after the deploy to a test environment, with three habits:

1. Secrets in a variable group. Keep credentials in Pipelines → Library as a variable group, mark the password secret, and link the group to the pipeline. The YAML refers to $(ShiftLeftPassword) and never contains it.
2. Publish results whatever happens. Give the publish step condition: succeededOrFailed(), so a failed gate still records which tests failed.
3. Fail the stage, not just the report. The test task's own exit code is the gate. A pipeline whose publish step merely shows red tests while the stage succeeds will deploy a broken API.

In Shift-Left API: the Azure Pipelines task

Shift-Left API publishes an Azure DevOps extension, Shift-Left API Automation Integration, on the Visual Studio Marketplace (publisher TotalShiftLeft). Its task, Shift-Left API: Run test pack, runs a Test Run Pack, applies a pass-rate gate and writes JUnit XML. Test Run Packs and CI need the Professional or Enterprise edition, or the 15-day trial.

1. Install the extension from the Visual Studio Marketplace into your Azure DevOps organization. The Marketplace has version 1.0.0; the GitHub releases of Total-Shift-Left/Shift-Left-API-Integrations carry a newer 1.0.1 VSIX.
2. Make the pack. In Shift-Left API, build a Test Run Pack of sandbox read tests in Execution and copy its ID. Make sure the API user's role is allowed under Settings → Integrations → Public API.
3. Add a variable group named shiftleft-secrets with ShiftLeftEmail and a secret ShiftLeftPassword.
4. Add the steps to your pipeline:

variables:
  - group: shiftleft-secrets   # ShiftLeftEmail, ShiftLeftPassword (secret)

steps:
  - task: totalshiftleft.shiftleft-api-integration-task@1
    name: apiTests
    displayName: Run the Store API pack
    inputs:
      serverUrl: 'https://your-shiftleft-host'
      apiEmail: '$(ShiftLeftEmail)'
      apiPassword: '$(ShiftLeftPassword)'
      packId: 'your_pack_id'
      passThresholdPercent: '90'
      failOnErrorTests: true
      gateFailureResult: 'failed'
      writeTestResultsXml: true
      testResultsXmlPath: 'shiftleft-test-pack-results.xml'

  - task: PublishTestResults@2
    displayName: Publish API test results
    condition: succeededOrFailed()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: '**/shiftleft-test-pack-results.xml'
      searchFolder: '$(Build.SourcesDirectory)'

5. Run the pipeline. The task signs in to the public API under /api/v1, starts the pack, polls every 10 seconds (for up to 60 minutes by default) until the run finishes, and applies the gate.

The visible result: the task's log shows the decision PASSED, and the run's Tests tab lists each test in the pack with its outcome. Break one test in the pack and run again: four tests with one failing is 75 percent, under 90, so the decision is GATE_FAIL_THRESHOLD, the task fails, and the Tests tab shows which test broke.

Details worth knowing:

• Set writeTestResultsXml: true explicitly. Version 1.0.0 leaves it off by default, and without it there is no file to publish.
• Warn instead of block. gateFailureResult: 'succeededWithIssues' finishes the task as Succeeded with issues when the gate fails, useful while you tune a new pack.
• Output variables. With name: apiTests, later steps can read $(apiTests.shiftLeftDecision) and $(apiTests.shiftLeftExecutionId).
• Where the tests run. The pack runs on the Shift-Left API server, not on the Azure Pipelines agent. A Microsoft-hosted agent works when that server is reachable from the internet; otherwise use a self-hosted agent inside your network. The agent needs the Node 20 task handler.
• One environment per pack. To test dev and staging, run two packs; there's no per-run environment override.

The Azure DevOps integration page lists every input. Lesson 9.15 shows the same gate across all three CI tools in one demo.

Common mistakes

• Two copies of every case. The Test Plans case and the automated test drift apart. Pick one source of truth.
• Publishing only on success. Without succeededOrFailed(), a failed gate hides the results you need.
• A PAT with full access in a script. Scope tokens to the API areas the script uses, such as work items and test management, and give them an expiry.
• One shared environment for every pipeline. Two runs editing the same records make each other fail. Give CI its own environment.

Try it yourself

On a trial, install Shift-Left API Automation Integration from the Visual Studio Marketplace, add a shiftleft-secrets variable group, and run the YAML above with your server address and sandbox pack ID. Open the run's Tests tab: each test in the pack is listed with its outcome, and the task's log shows PASSED.

Recap

• Azure DevOps API test cases live in Test Plans as work items and in Pipelines as automated tests; keep one of them as the source of truth.
• Azure DevOps REST API test cases are scriptable: the Work Item Tracking API creates them, the Test Plans API organizes them, and PublishTestResults@2 records pipeline results.
• The Shift-Left API task runs a Test Run Pack from Azure Pipelines with a pass-rate gate and JUnit results for the Tests tab.

For multi-stage pipelines, deployment gates and test analytics, read Azure DevOps API testing: complete pipeline guide. Next, in lesson 7.5, you turn packs like this one into a nightly regression run.