API testing in Bitbucket Pipelines

bitbucket-pipelines.yml example

Drop this into your bitbucket-pipelines.yml. Runs on every pull request and on main-branch merges, and fails the step when the pass rate drops below 95%.

image: atlassian/default-image:4

pipelines:
  pull-requests:
    '**':
      - step:
          name: Shift-Left API Tests
          script:
            - apt-get update -qq && apt-get install -y -qq jq
            - |
                set -euo pipefail
                PACK="$SHIFTLEFT_TEST_PACK_ID"
                TOKEN=$(curl -sS -f -X POST "$SHIFTLEFT_URL/api/v1/login" \
                  -H "Content-Type: application/json" \
                  -d "{\"email\":\"$SHIFTLEFT_EMAIL\",\"password\":\"$SHIFTLEFT_PASSWORD\"}" \
                  | jq -r .token)
                AUTH="Authorization: Bearer $TOKEN"
                STATUS_URL="$SHIFTLEFT_URL/api/v1/test-packs/$PACK/status"

                # Note the pack's latest run first, so the previous run is never graded
                BEFORE=$(curl -sS -f "$STATUS_URL" -H "$AUTH" | jq -r '.executionId // empty')
                # Start the pack (the server answers 409 if it is already running)
                curl -sS -f -X POST "$SHIFTLEFT_URL/api/v1/test-packs/$PACK/run" -H "$AUTH" > /dev/null

                # Wait up to 60 minutes for a NEW run to finish
                DONE=0
                for i in $(seq 1 360); do
                  RESULT=$(curl -sS -f "$STATUS_URL" -H "$AUTH")
                  RUN_ID=$(echo "$RESULT" | jq -r '.executionId // empty')
                  STATE=$(echo "$RESULT" | jq -r '.status // empty' | tr '[:lower:]' '[:upper:]')
                  if [ -n "$RUN_ID" ] && [ "$RUN_ID" != "$BEFORE" ]; then
                    case "$STATE" in RUNNING|PENDING|QUEUED|"") ;; *) DONE=1; break ;; esac
                  fi
                  sleep 10
                done
                [ "$DONE" = 1 ] || { echo "Timed out waiting for a new run"; exit 1; }

                echo "$RESULT" | jq .
                # Reads executionId, status and successRate; if your installation's Swagger page
                # names them differently, adjust the jq filters. Fails closed below 95%.
                echo "$RESULT" | jq -e '(.successRate // 0) >= 95' > /dev/null

  branches:
    main:
      - step:
          name: Shift-Left API Tests (main)
          script:
            - apt-get update -qq && apt-get install -y -qq jq
            - |
                set -euo pipefail
                PACK="$SHIFTLEFT_TEST_PACK_ID"
                TOKEN=$(curl -sS -f -X POST "$SHIFTLEFT_URL/api/v1/login" \
                  -H "Content-Type: application/json" \
                  -d "{\"email\":\"$SHIFTLEFT_EMAIL\",\"password\":\"$SHIFTLEFT_PASSWORD\"}" \
                  | jq -r .token)
                AUTH="Authorization: Bearer $TOKEN"
                STATUS_URL="$SHIFTLEFT_URL/api/v1/test-packs/$PACK/status"

                # Note the pack's latest run first, so the previous run is never graded
                BEFORE=$(curl -sS -f "$STATUS_URL" -H "$AUTH" | jq -r '.executionId // empty')
                # Start the pack (the server answers 409 if it is already running)
                curl -sS -f -X POST "$SHIFTLEFT_URL/api/v1/test-packs/$PACK/run" -H "$AUTH" > /dev/null

                # Wait up to 60 minutes for a NEW run to finish
                DONE=0
                for i in $(seq 1 360); do
                  RESULT=$(curl -sS -f "$STATUS_URL" -H "$AUTH")
                  RUN_ID=$(echo "$RESULT" | jq -r '.executionId // empty')
                  STATE=$(echo "$RESULT" | jq -r '.status // empty' | tr '[:lower:]' '[:upper:]')
                  if [ -n "$RUN_ID" ] && [ "$RUN_ID" != "$BEFORE" ]; then
                    case "$STATE" in RUNNING|PENDING|QUEUED|"") ;; *) DONE=1; break ;; esac
                  fi
                  sleep 10
                done
                [ "$DONE" = 1 ] || { echo "Timed out waiting for a new run"; exit 1; }

                echo "$RESULT" | jq .
                # Reads executionId, status and successRate; if your installation's Swagger page
                # names them differently, adjust the jq filters. Fails closed below 95%.
                echo "$RESULT" | jq -e '(.successRate // 0) >= 95' > /dev/null

The pipe source is public and wraps @totalshiftleft/ci — the single implementation behind the GitHub Action, the Azure DevOps task, the CircleCI orb and the GitLab component, so it grades a run identically to all of them. The image is not on Docker Hub yet, so the snippet below is what the step will look like, not something you can run today.

pipelines:
  default:
    - step:
        name: API tests
        script:
          - pipe: totalshiftleft/shiftleft-test-pack:1.0.0
            variables:
              SHIFTLEFT_SERVER_URL: 'https://tenant.totalshiftleft.ai'
              SHIFTLEFT_API_EMAIL: $SHIFTLEFT_API_EMAIL
              SHIFTLEFT_API_PASSWORD: $SHIFTLEFT_API_PASSWORD
              SHIFTLEFT_TEST_PACK_ID: 'pack_123'
              SHIFTLEFT_PASS_THRESHOLD_PERCENT: '95'
        artifacts:
          - shiftleft-test-pack-summary.json
          - shiftleft-test-pack-results.xml

Define SHIFTLEFT_API_EMAIL and SHIFTLEFT_API_PASSWORD as secured repository, deployment or workspace variables so they are masked in the log. Bitbucket picks JUnit XML up automatically from a few well-known locations, so setting SHIFTLEFT_TEST_RESULTS_XML_PATH to test-results/shiftleft.xml puts the run in the Tests tab.

Gate decision codes

Printed to the step log as [ShiftLeft] Output decision=<code> and repeated in the JSON summary artifact. The step exits 0 when the gate passes and 1 when it fails, so it gates the build.