Azure DevOps integration

Azure DevOps Integration for API Testing

The Shift-Left API Azure DevOps task is a first-party integration for Azure Pipelines. Install it from the Visual Studio Marketplace or as a VSIX; one YAML task then triggers a test pack, applies a pass-rate and fail-on-error quality gate, writes JUnit XML and a JSON summary, and sets output variables for later steps.

Overview

The Shift-Left API Automation Integration extension for Azure DevOps adds a pipeline task that connects to your Shift-Left API instance using the public API. It handles login, test pack triggering, status polling, and detailed results — all within your existing Azure Pipeline workflow.

Prerequisites

  • Node.js 20+ on the build agent (task handler Node20_1).
  • Network access from the agent to your Shift-Left API server URL.
  • A user with permission to call the Shift-Left CI/CD public API.

Key features

  • Trigger test run packs from any YAML or Classic pipeline.
  • Configurable quality gate with pass-rate thresholds and error-test policies.
  • Automatic JUnit XML publishing for native Azure DevOps test result tabs.
  • JSON summary artifact for custom dashboards or downstream steps.
  • Output variables (execution ID, trigger ID, gate decision) for pipeline logic.
  • Multi-tenant support via optional X-Tenant-ID header.

Pipeline YAML example

Reference the task as totalshiftleft.shiftleft-api-integration-task@1 in your pipeline YAML.

azure-pipelines.yml
variables:
  - group: shiftleft-secrets   # contains ShiftLeftEmail, ShiftLeftPassword

steps:
  - task: totalshiftleft.shiftleft-api-integration-task@1
    displayName: Run ShiftLeft pack
    inputs:
      serverUrl: 'https://your-shiftleft-host'
      tenantId: ''                    # optional
      apiEmail: '$(ShiftLeftEmail)'
      apiPassword: '$(ShiftLeftPassword)'
      packId: 'your_pack_id'
      waitForCompletion: true
      pollIntervalSeconds: '10'
      timeoutMinutes: '60'
      passThresholdPercent: '100'
      failOnErrorTests: true
      gateFailureResult: 'failed'
      writeJsonSummary: true
      jsonSummaryPath: 'shiftleft-test-pack-summary.json'
      writeTestResultsXml: true
      testResultsXmlPath: 'shiftleft-test-pack-results.xml'
      workingDirectory: '$(Build.SourcesDirectory)'

  - task: PublishTestResults@2
    displayName: Publish ShiftLeft test results
    condition: succeededOrFailed()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: '**/shiftleft-test-pack-results.xml'
      searchFolder: '$(Build.SourcesDirectory)'
      mergeTestResults: true
      failTaskOnFailedTests: false

  - task: PublishPipelineArtifact@1
    displayName: Upload JSON summary
    condition: always()
    inputs:
      targetPath: '$(Build.SourcesDirectory)/shiftleft-test-pack-summary.json'
      artifact: 'shiftleft-summary'

Configuration reference

InputRequiredDescription
serverUrlRequiredBase URL of your Shift-Left API instance.
tenantIdOptionalOptional tenant ID for multi-tenant environments. Sends X-Tenant-ID header.
apiEmailRequiredLogin email for Shift-Left API. Use a secret variable.
apiPasswordRequiredLogin password. Must be a secret pipeline variable.
packIdRequiredID of the test run pack to execute.
waitForCompletionOptionalPoll until the run finishes or times out. Default: true.
pollIntervalSecondsOptionalSeconds between status polls. Default: 10.
timeoutMinutesOptionalOverall timeout in minutes. Default: 60.
passThresholdPercentOptionalMinimum pass rate (0–100) for a successful gate. Default: 100.
failOnErrorTestsOptionalFail the gate when any test has ERROR status. Default: true.
gateFailureResultOptionalTask result on gate failure: failed (default) or succeeded-with-issues, which finishes the task as Succeeded with issues.
writeJsonSummaryOptionalWrite a JSON summary file to the workspace. Default: true.
jsonSummaryPathOptionalWorkspace-relative path for the JSON summary file.
writeTestResultsXmlOptionalWrite JUnit XML results to the workspace. Default: true.
testResultsXmlPathOptionalWorkspace-relative path for the JUnit XML file.

Output variables

Give the task a name: (for example ShiftLeft) and read these variables in downstream steps.

VariableDescription
shiftLeftExecutionIdExecution ID from the status/results response.
shiftLeftTriggerExecutionIdID returned from the initial trigger response.
shiftLeftDecisionGate decision code: PASSED, GATE_FAIL_THRESHOLD, TIMEOUT, etc.

Quality gate mapping

How the task sets the Azure Pipelines step outcome based on your gate configuration.

Task resultMeaning
SucceededGate passed — run completed as expected.
FailedGate or run failed. Requires gateFailureResult set to failed.
Succeeded with issuesGate failed but configured as non-blocking (gateFailureResult: succeeded-with-issues).

Step-by-step setup, every input and troubleshooting: Run test packs from Azure DevOps Pipelines (help centre). For the tool-neutral how-to on running API tests in Azure Pipelines, read API test automation with CI/CD, step by step; to run security or load tests from the same packs, see API security testing and API load testing (paid add-ons).

Generated pipeline file

A ready-made Azure Pipelines file from a test pack

On a test run pack, Run in your pipeline writes a ready-made Azure Pipelines file for you. It has no field for a password or token, so nothing you type is pasted into the file unchecked. Studio generates the same kind of file for GitHub Actions, GitLab CI, Jenkins.

  • +Where the runner comes from. Keep the runner in your repository, or give an https address and your own SHA-256 checksum. If the checksum does not match or the download fails, the build stops before the runner starts. The runner needs Node 18 or newer and has no npm dependency.
  • +Options the file can carry. --verdict: Fail the build on the release verdict. A verdict of "no evidence yet" fails the build by default, and a partial verdict fails unless you allow it. --tickets: Run only the tests behind the tickets you name; a ticket with no tests fails the build unless you allow that. --sarif, --fail-on-severity, --policy: Options for security runs: SARIF output, a severity gate and a committed policy file. These need the API security testing add-on.
  • +Pull request comments. The verdict comment appears on Azure Pipelines pull requests.
  • +Plans. Professional, Trial and Enterprise. The Citizen Developer edition cannot generate pipeline files.

See the release verdict for what --verdict judges.

Azure DevOps FAQs

Contact us at

info@totalshiftleft.com

to learn more

  • How do I reference the task in YAML?

    Use totalshiftleft.shiftleft-api-integration-task@1 (publisher ID + task name + major version).

  • Where should I store credentials?

    Create a variable group in Pipelines → Library with ShiftLeftEmail (plain) and ShiftLeftPassword (secret). Link the group to your pipeline.

  • What Node.js version does the agent need?

    The task handler requires Node.js 20 or later on the build agent (Node20_1 handler).

  • Can I use this with YAML and Classic pipelines?

    Yes. The YAML task reference works in both YAML pipelines and Classic release definitions that support YAML tasks.

Ready to add API testing to your Azure Pipeline?

Start your free 15-day trial. Install the extension and run your first test pack today.

Updated