CircleCI Integration for API Testing

.circleci/config.yml example

Drop this into your .circleci/config.yml. References a Context for credentials, runs on main and release branches, and fails the job when the pass rate drops below 95%.

version: 2.1

jobs:
  api-tests:
    docker:
      - image: cimg/base:2024.01
    steps:
      - run:
          name: Run Shift-Left API test pack via REST API
          command: |
            set -euo pipefail
            PACK="$SHIFTLEFT_TEST_PACK_ID"
            TOKEN=$(curl -sS -f -X POST "$SHIFTLEFT_URL/api/v1/login" \
              -H "Content-Type: application/json" \
              -d "{\"email\":\"$SHIFTLEFT_EMAIL\",\"password\":\"$SHIFTLEFT_PASSWORD\"}" \
              | jq -r .token)
            AUTH="Authorization: Bearer $TOKEN"
            STATUS_URL="$SHIFTLEFT_URL/api/v1/test-packs/$PACK/status"

            # Note the pack's latest run first, so the previous run is never graded
            BEFORE=$(curl -sS -f "$STATUS_URL" -H "$AUTH" | jq -r '.executionId // empty')
            # Start the pack (the server answers 409 if it is already running)
            curl -sS -f -X POST "$SHIFTLEFT_URL/api/v1/test-packs/$PACK/run" -H "$AUTH" > /dev/null

            # Wait up to 60 minutes for a NEW run to finish
            DONE=0
            for i in $(seq 1 360); do
              RESULT=$(curl -sS -f "$STATUS_URL" -H "$AUTH")
              RUN_ID=$(echo "$RESULT" | jq -r '.executionId // empty')
              STATE=$(echo "$RESULT" | jq -r '.status // empty' | tr '[:lower:]' '[:upper:]')
              if [ -n "$RUN_ID" ] && [ "$RUN_ID" != "$BEFORE" ]; then
                case "$STATE" in RUNNING|PENDING|QUEUED|"") ;; *) DONE=1; break ;; esac
              fi
              sleep 10
            done
            [ "$DONE" = 1 ] || { echo "Timed out waiting for a new run"; exit 1; }

            echo "$RESULT" | jq .
            # Reads executionId, status and successRate; if your installation's Swagger page
            # names them differently, adjust the jq filters. Fails closed below 95%.
            echo "$RESULT" | jq -e '(.successRate // 0) >= 95' > /dev/null

workflows:
  test:
    jobs:
      - api-tests:
          context: shift-left-credentials
          filters:
            branches:
              only:
                - main
                - /^release\/.*/

The orb source is public and wraps @totalshiftleft/ci — the single implementation behind the GitHub Action, the Azure DevOps task, the Bitbucket pipe and the GitLab component, so it grades a run identically to all of them. It is not on the CircleCI registry yet, so the snippet below is what the install will look like, not something you can run today.

version: 2.1

orbs:
  shiftleft: totalshiftleft/shiftleft@1.0.0

workflows:
  api-tests:
    jobs:
      - shiftleft/api-tests:
          context: shiftleft
          server-url: https://tenant.totalshiftleft.ai
          pack-id: pack_123
          pass-threshold-percent: 95

Credentials are passed by variable name, never by value: the api-email and api-password parameters default to SHIFTLEFT_API_EMAIL and SHIFTLEFT_API_PASSWORD in your context, so no secret is written into .circleci/config.yml.

Gate decision codes

CircleCI has no step-output mechanism, so the orb prints these to the job log as [ShiftLeft] Output decision=<code> and repeats them in the JSON summary artifact.