API testing in Bitbucket Pipelines

bitbucket-pipelines.yml example

Drop this into your bitbucket-pipelines.yml. Runs on every pull request and on main-branch merges, and fails the step when the pass rate drops below 95%.

image: atlassian/default-image:4

pipelines:
  pull-requests:
    '**':
      - step:
          name: Total Shift Left Tests
          script:
            - apt-get update -qq && apt-get install -y -qq jq
            - TOKEN=$(curl -s -X POST "$SHIFTLEFT_URL/api/v1/auth/login" \
                -H "Content-Type: application/json" \
                -d "{\"email\":\"$SHIFTLEFT_EMAIL\",\"password\":\"$SHIFTLEFT_PASSWORD\"}" \
                | jq -r .token)
            - EXEC_ID=$(curl -s -X POST "$SHIFTLEFT_URL/api/v1/test-packs/$SHIFTLEFT_TEST_PACK_ID/run" \
                -H "Authorization: Bearer $TOKEN" | jq -r .executionId)
            - |
                # Poll until the run has a result, up to 2 minutes
                for i in $(seq 1 24); do
                  RESULT=$(curl -s "$SHIFTLEFT_URL/api/v1/executions/$EXEC_ID" -H "Authorization: Bearer $TOKEN")
                  [ "$(echo "$RESULT" | jq -r '.passRate != null')" = "true" ] && break
                  sleep 5
                done
                echo "$RESULT" | jq .
                # Fails closed: a missing/null/non-numeric passRate (auth error, bad execution
                # id, run still in progress after the timeout above) is a failure, not a pass.
                echo "$RESULT" | jq -e '(.passRate // 0) >= 95' > /dev/null

  branches:
    main:
      - step:
          name: Total Shift Left Tests (main)
          script:
            - apt-get update -qq && apt-get install -y -qq jq
            - TOKEN=$(curl -s -X POST "$SHIFTLEFT_URL/api/v1/auth/login" \
                -H "Content-Type: application/json" \
                -d "{\"email\":\"$SHIFTLEFT_EMAIL\",\"password\":\"$SHIFTLEFT_PASSWORD\"}" \
                | jq -r .token)
            - EXEC_ID=$(curl -s -X POST "$SHIFTLEFT_URL/api/v1/test-packs/$SHIFTLEFT_TEST_PACK_ID/run" \
                -H "Authorization: Bearer $TOKEN" | jq -r .executionId)
            - |
                # Poll until the run has a result, up to 2 minutes
                for i in $(seq 1 24); do
                  RESULT=$(curl -s "$SHIFTLEFT_URL/api/v1/executions/$EXEC_ID" -H "Authorization: Bearer $TOKEN")
                  [ "$(echo "$RESULT" | jq -r '.passRate != null')" = "true" ] && break
                  sleep 5
                done
                echo "$RESULT" | jq .
                # Fails closed: a missing/null/non-numeric passRate (auth error, bad execution
                # id, run still in progress after the timeout above) is a failure, not a pass.
                echo "$RESULT" | jq -e '(.passRate // 0) >= 95' > /dev/null

The pipe source is public and wraps @totalshiftleft/ci — the single implementation behind the GitHub Action, the Azure DevOps task, the CircleCI orb and the GitLab component, so it grades a run identically to all of them. The image is not on Docker Hub yet, so the snippet below is what the step will look like, not something you can run today.

pipelines:
  default:
    - step:
        name: API tests
        script:
          - pipe: totalshiftleft/shiftleft-test-pack:1.0.0
            variables:
              SHIFTLEFT_SERVER_URL: 'https://tenant.totalshiftleft.ai'
              SHIFTLEFT_API_EMAIL: $SHIFTLEFT_API_EMAIL
              SHIFTLEFT_API_PASSWORD: $SHIFTLEFT_API_PASSWORD
              SHIFTLEFT_PACK_ID: 'pack_123'
              SHIFTLEFT_PASS_THRESHOLD_PERCENT: '95'
        artifacts:
          - shiftleft-test-pack-summary.json
          - shiftleft-test-pack-results.xml

Define SHIFTLEFT_API_EMAIL and SHIFTLEFT_API_PASSWORD as secured repository, deployment or workspace variables so they are masked in the log. Bitbucket picks JUnit XML up automatically from a few well-known locations, so setting SHIFTLEFT_TEST_RESULTS_XML_PATH to test-results/shiftleft.xml puts the run in the Tests tab.

Gate decision codes

Printed to the step log as [ShiftLeft] Output decision=<code> and repeated in the JSON summary artifact. The step exits 0 when the gate passes and 1 when it fails, so it gates the build.