CI/CD: Run Test Packs from Azure DevOps Pipelines

Overview

The Shift-Left API task for Azure Pipelines runs one of your test run packs as a pipeline step. It signs in to your Shift-Left API server, starts the pack, waits for that run to finish, applies a quality gate, and writes JUnit XML and a JSON summary into the working directory. If the gate fails, the task fails (or, if you choose, finishes as succeeded with issues).

The task applies the same gate as the GitHub Action and the Jenkins plugin, so the same run gets the same decision everywhere.

Before you begin

1. A test run pack with its environment chosen in the pack. Copy the pack ID. See Test Run Pack Wizard.
2. The Public API switched on, with the CI user's role in Allowed Roles, under Settings → Integrations → Public API. See Public API.
3. A CI user (email and password). The task signs in with these, not with an API key.
4. A server the agent can reach. The pack runs on your Shift-Left API server. For a self-hosted installation inside your network, use a self-hosted Azure Pipelines agent that can reach it.

Step 1 — Install the extension

• Azure DevOps Services: install Shift-Left API Automation Integration from the Visual Studio Marketplace into your organisation.
• Azure DevOps Server (not verified), or no Marketplace access: download the .vsix from the azure-devops release in Total-Shift-Left/Shift-Left-API-Integrations and upload it under Collection Settings → Extensions. The GitHub release can be newer than the Marketplace copy.

Step 2 — Store the credentials

Create a variable group (for example shiftleft-secrets) under Pipelines → Library with ShiftLeftEmail and ShiftLeftPassword, and mark the password as secret. Put the server URL and pack ID in pipeline variables:

variables:
  - group: shiftleft-secrets      # ShiftLeftEmail, ShiftLeftPassword
  - name: SHIFTLEFT_URL
    value: 'https://shiftleft.yourcompany.com'
  - name: SHIFTLEFT_TEST_PACK_ID
    value: 'your_pack_id'

Step 3 — Add the task

Reference the task as totalshiftleft.shiftleft-api-integration-task@1:

- task: totalshiftleft.shiftleft-api-integration-task@1
  displayName: 'Run API Tests'
  inputs:
    serverUrl: $(SHIFTLEFT_URL)
    apiEmail: $(ShiftLeftEmail)
    apiPassword: $(ShiftLeftPassword)
    packId: $(SHIFTLEFT_TEST_PACK_ID)
    passThresholdPercent: '100'
    failOnErrorTests: true
    gateFailureResult: 'failed'      # or 'succeeded-with-issues' to warn instead
    testResultsXmlPath: 'test-results/shiftleft-test-pack-results.xml'

Then publish the JUnit file so results show on the pipeline's Tests tab:

- task: PublishTestResults@2
  displayName: 'Publish API Test Results'
  condition: succeededOrFailed()
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/test-results/*.xml'
    mergeTestResults: true
    testRunTitle: 'API Tests - $(Build.BuildNumber)'

Inputs

The task reports the same gate decisions as every integration: PASSED, GATE_FAIL_THRESHOLD, GATE_FAIL_ERROR_TESTS, FAILED, COMPLETED_WITH_ISSUES, OK, TIMEOUT and TRIGGER_ONLY. See the decision table for what each means.

Several packs in one stage

To run smoke, regression and contract packs side by side, use a matrix and give each job its own pack ID:

- stage: APITest
  jobs:
    - job: RunAPITests
      strategy:
        matrix:
          smoke_tests:
            TEST_SUITE: 'smoke'
            PACK_ID: 'pack_smoke'
          regression_tests:
            TEST_SUITE: 'regression'
            PACK_ID: 'pack_regression'
          contract_tests:
            TEST_SUITE: 'contract'
            PACK_ID: 'pack_contract'
      steps:
        - task: totalshiftleft.shiftleft-api-integration-task@1
          inputs:
            serverUrl: $(SHIFTLEFT_URL)
            apiEmail: $(ShiftLeftEmail)
            apiPassword: $(ShiftLeftPassword)
            packId: $(PACK_ID)
            testResultsXmlPath: 'test-results/shiftleft-test-pack-results.xml'
        - task: PublishTestResults@2
          inputs:
            testResultsFormat: 'JUnit'
            testResultsFiles: '**/test-results/*.xml'
            testRunTitle: '$(TEST_SUITE) - $(Build.BuildNumber)'
          condition: always()

Good to know

• The environment comes from the pack. There is no per-run environment override. Keep one pack per environment and choose the pack ID per stage or branch.
• One run per pack at a time. A second start of a running pack is refused (HTTP 409), so parallel jobs need different packs.
• It grades the run it started, not the pack's previous run.
• No pull-request comment. The pipeline check passes or fails, and the JUnit results appear on the Tests tab.
• CI sign-ins and CI-started runs are recorded in the audit log.

Related articles

• CI/CD integrations on the product site
• Azure DevOps integration overview
• GitHub Actions
• Any CI via the REST API
• Public API
• Plans and editions