CI/CD: Any CI via the REST API (GitLab, CircleCI, Bitbucket and others)

Overview

Shift-Left API has first-party integrations for GitHub Actions, Azure DevOps and Jenkins. Every other CI system (GitLab CI, CircleCI, Bitbucket Pipelines, TeamCity, Bamboo and the rest) runs a pack through the Public API at <your server>/api/v1. A GitLab component, a CircleCI orb and a Bitbucket pipe exist as source in the integrations repository but are not published to their registries yet, so this page uses plain curl.

The flow is the same one the first-party integrations follow:

1. POST /api/v1/login with the CI user's email and password returns a token. Send it as Authorization: Bearer <token> on every other call.
2. GET /api/v1/test-packs/{packId}/status before you start, to note the pack's latest run.
3. POST /api/v1/test-packs/{packId}/run starts the pack. The server answers 409 if that pack is already running.
4. GET /api/v1/test-packs/{packId}/status until a new run has finished, then compare its pass rate with your threshold.
5. Optionally, GET /api/v1/test-packs/{packId}/results?executionId=<id> for the per-test results.

Step 2 matters. The status endpoint reports the pack's most recent run, and the run you just started may not have been picked up yet, so a script that polls straight away can read the previous run and pass a build whose new run has not begun. Wait until the run id changes.

Before you begin

• A test run pack with its environment chosen in the pack, and its pack ID. The API has no per-run environment or base-URL override; keep one pack per environment.
• The Public API switched on, with the CI user's role in Allowed Roles (Settings → Integrations → Public API). The same screen links to the API's Swagger page, which shows the exact request and response shapes for your version.
• The CI user's email and password stored as masked or secret CI variables. You can send an API key (sk_live_…) as the Bearer token instead of signing in.
• curl and jq on the CI image.

The script

Save this as ci/run-shiftleft-pack.sh in your repository and make it executable. It exits non-zero when the run fails, the pass rate is below the threshold, or the wait times out.

#!/usr/bin/env bash
# Run a Shift-Left API test pack and fail below a pass-rate threshold.
# Needs curl and jq. Reads: SHIFTLEFT_URL, SHIFTLEFT_EMAIL, SHIFTLEFT_PASSWORD,
# SHIFTLEFT_TEST_PACK_ID, and optionally SHIFTLEFT_PASS_THRESHOLD (default 100)
# and SHIFTLEFT_TIMEOUT_MINUTES (default 60).
set -euo pipefail

API="$SHIFTLEFT_URL/api/v1"
PACK="$SHIFTLEFT_TEST_PACK_ID"
THRESHOLD="${SHIFTLEFT_PASS_THRESHOLD:-100}"
DEADLINE=$(( $(date +%s) + ${SHIFTLEFT_TIMEOUT_MINUTES:-60} * 60 ))

# 1. Sign in and keep the token.
TOKEN=$(curl -sS -f -X POST "$API/login" -H "Content-Type: application/json" \
  -d "$(jq -n --arg e "$SHIFTLEFT_EMAIL" --arg p "$SHIFTLEFT_PASSWORD" '{email: $e, password: $p}')" \
  | jq -r '.token')
AUTH="Authorization: Bearer $TOKEN"

# 2. Note the latest run before starting, so the previous run is never graded.
# A pack that has never run has no latest run; "|| true" keeps that from stopping the script.
BEFORE=$(curl -sS "$API/test-packs/$PACK/status" -H "$AUTH" | jq -r '.executionId // empty' || true)

# 3. Start the pack. 409 means it is already running.
CODE=$(curl -sS -o /dev/null -w '%{http_code}' -X POST "$API/test-packs/$PACK/run" -H "$AUTH")
if [ "$CODE" = "409" ]; then echo "Pack $PACK is already running"; exit 1; fi
if [ "$CODE" -lt 200 ] || [ "$CODE" -ge 300 ]; then echo "Could not start pack $PACK (HTTP $CODE)"; exit 1; fi

# 4. Wait for a new run to finish.
while :; do
  STATUS=$(curl -sS -f "$API/test-packs/$PACK/status" -H "$AUTH")
  RUN_ID=$(echo "$STATUS" | jq -r '.executionId // empty')
  STATE=$(echo "$STATUS" | jq -r '.status // empty' | tr '[:lower:]' '[:upper:]')
  if [ -n "$RUN_ID" ] && [ "$RUN_ID" != "$BEFORE" ]; then
    case "$STATE" in RUNNING|PENDING|QUEUED|"") ;; *) break ;; esac
  fi
  if [ "$(date +%s)" -ge "$DEADLINE" ]; then echo "Timed out waiting for pack $PACK"; exit 1; fi
  sleep 10
done

# 5. Gate on the pass rate.
RATE=$(echo "$STATUS" | jq -r '.successRate // 0')
echo "Run $RUN_ID finished: $STATE, pass rate $RATE% (threshold $THRESHOLD%)"
echo "$STATUS" | jq -e --argjson t "$THRESHOLD" '(.successRate // 0) >= $t' > /dev/null

The script reads three fields from the status response: executionId, status and successRate. If the Swagger page for your installation names or nests them differently, change those jq filters to match.

GitLab CI

# .gitlab-ci.yml
api-tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache bash curl jq
  script:
    - bash ci/run-shiftleft-pack.sh
  variables:
    SHIFTLEFT_PASS_THRESHOLD: "95"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Set SHIFTLEFT_URL, SHIFTLEFT_EMAIL, SHIFTLEFT_PASSWORD (masked) and SHIFTLEFT_TEST_PACK_ID under Settings → CI/CD → Variables.

CircleCI

# .circleci/config.yml
version: 2.1
jobs:
  api-tests:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run: sudo apt-get update && sudo apt-get install -y jq
      - run: bash ci/run-shiftleft-pack.sh
workflows:
  test:
    jobs:
      - api-tests:
          context: shiftleft

Keep the four SHIFTLEFT_* values in a context (here shiftleft) or in the project's environment variables.

Bitbucket Pipelines

# bitbucket-pipelines.yml
pipelines:
  pull-requests:
    '**':
      - step:
          name: API tests
          image: alpine:latest
          script:
            - apk add --no-cache bash curl jq
            - bash ci/run-shiftleft-pack.sh

Add the four SHIFTLEFT_* values as secured repository variables.

Good to know

• Where the tests run. The pack runs on your Shift-Left API server, not on the CI runner. For a self-hosted installation inside your network, the CI runner only needs to reach the server, and the server needs to reach the API under test.
• No JUnit from this script. The first-party integrations write JUnit XML for you. With the REST API, fetch GET /api/v1/test-packs/{packId}/results?executionId=<id> and convert it if your CI needs a test report.
• Scopes. The /api/v1 endpoints check that the caller is signed in; they do not check API-key scopes. Any valid key or token can list and run packs, so give CI its own user or key and revoke it when it is no longer needed.
• Rate limits are per IP address, not per key.
• Audit. CI sign-ins (cicd_login) and CI-started runs (cicd_execute) are recorded in the audit log.
• Load tests use a different API. See "Automating this" in Create and Run a Performance Test.

Related articles

• CI/CD integrations on the product site
• GitLab CI integration overview
• Public API
• API keys
• GitHub Actions
• Plans and editions