CircleCI integration

API testing in CircleCI

The first-party CircleCI orb is public in source and awaiting registry publication. Total Shift Left's public REST API lets you trigger test packs, gate workflows on pass-rate thresholds, and pull results into any pipeline today.

Book DemoAll integrations

The Total Shift Left CircleCI orb is not on the CircleCI registry yet, so totalshiftleft/shiftleft@1.0.0 will not resolve today. Its source is public — see circleci-orb/. Everything below is a real, working pattern using the public REST API directly.

What this integration gives you

The job below runs a Total Shift Left test pack as part of any CircleCI workflow, using nothing but curl and jq against the public REST API: authenticate, trigger a run, and check the result. Use it to gate workflows on API quality, run scheduled regression runs, or chain into broader release pipelines.

  • Quality gates on workflows

    Exit non-zero when the pass rate misses your threshold — fails the job and blocks downstream steps.

  • JSON results

    The execution endpoint returns structured JSON; pipe it into your own reporting or artifact upload step.

  • Context-secure credentials

    Use CircleCI Contexts to scope credentials to specific workflows. No project-level secrets to manage and rotate.

  • Self-hosted runner support

    Works with CircleCI cloud or your self-hosted runners — useful when your Shift-Left deployment is behind a firewall.

.circleci/config.yml example

Drop this into your .circleci/config.yml. References a Context for credentials, runs on main and release branches, and fails the job when the pass rate drops below 95%.

version: 2.1

jobs:
  api-tests:
    docker:
      - image: cimg/base:2024.01
    steps:
      - run:
          name: Run Total Shift Left test pack via REST API
          command: |
            TOKEN=$(curl -s -X POST "$SHIFTLEFT_URL/api/v1/auth/login" \
              -H "Content-Type: application/json" \
              -d "{\"email\":\"$SHIFTLEFT_EMAIL\",\"password\":\"$SHIFTLEFT_PASSWORD\"}" \
              | jq -r .token)

            EXEC_ID=$(curl -s -X POST "$SHIFTLEFT_URL/api/v1/test-packs/$SHIFTLEFT_TEST_PACK_ID/run" \
              -H "Authorization: Bearer $TOKEN" | jq -r .executionId)

            # Poll until the run has a result, up to 2 minutes
            for i in $(seq 1 24); do
              RESULT=$(curl -s "$SHIFTLEFT_URL/api/v1/executions/$EXEC_ID" -H "Authorization: Bearer $TOKEN")
              [ "$(echo "$RESULT" | jq -r '.passRate != null')" = "true" ] && break
              sleep 5
            done

            echo "$RESULT" | jq .
            # Fails closed: a missing/null/non-numeric passRate (auth error, bad execution id,
            # run still in progress after the timeout above) is treated as a failure, not a pass.
            echo "$RESULT" | jq -e '(.passRate // 0) >= 95' > /dev/null

workflows:
  test:
    jobs:
      - api-tests:
          context: shift-left-credentials
          filters:
            branches:
              only:
                - main
                - /^release\/.*/

Environment variables

Define these in a CircleCI Context (Organization Settings → Contexts) and reference the context from your workflow. Contexts give you central rotation, audit trail, and team-level access controls.

VariableDescription
SHIFTLEFT_URLBase URL of your Total Shift Left deployment. Define in a CircleCI Context for secure sharing across projects.
SHIFTLEFT_EMAIL / SHIFTLEFT_PASSWORDAuthentication credentials. Always use a CircleCI Context (org-scoped) — never project environment variables for credentials.
SHIFTLEFT_TEST_PACK_IDID of the test pack. Define per-project; reference from the workflow.

The orb, once it is published

Not yet installable

The orb source is public and wraps @totalshiftleft/ci — the single implementation behind the GitHub Action, the Azure DevOps task, the Bitbucket pipe and the GitLab component, so it grades a run identically to all of them. It is not on the CircleCI registry yet, so the snippet below is what the install will look like, not something you can run today.

version: 2.1

orbs:
  shiftleft: totalshiftleft/shiftleft@1.0.0

workflows:
  api-tests:
    jobs:
      - shiftleft/api-tests:
          context: shiftleft
          server-url: https://tenant.totalshiftleft.ai
          pack-id: pack_123
          pass-threshold-percent: 95

Credentials are passed by variable name, never by value: the api-email and api-password parameters default to SHIFTLEFT_API_EMAIL and SHIFTLEFT_API_PASSWORD in your context, so no secret is written into .circleci/config.yml.

Gate decision codes

CircleCI has no step-output mechanism, so the orb prints these to the job log as [ShiftLeft] Output decision=<code> and repeats them in the JSON summary artifact.

DecisionMeaning
PASSEDThe run finished and cleared every gate you configured.
GATE_FAIL_THRESHOLDThe run finished below your pass-rate threshold.
GATE_FAIL_ERROR_TESTSAt least one test finished in ERROR and fail-on-error-tests was on.
FAILEDThe execution itself failed.
COMPLETED_WITH_ISSUESThe gate failed but gate-failure-result was set to warn instead.
OKThe run completed with no gate configured to judge it.
TIMEOUTThe run had not finished within timeout-minutes.
TRIGGER_ONLYwait-for-completion was false, so nothing was graded.

CircleCI integration — FAQ

Contact us at

info@totalshiftleft.com

to learn more

  • Is there a first-party Total Shift Left CircleCI orb?
    The orb exists in source — it lives in the public Shift-Left-API-Integrations repository and wraps the same shared runner as every other Total Shift Left CI integration. It has not been published to the CircleCI orb registry yet, so `orbs: shiftleft: totalshiftleft/shiftleft@1.0.0` will not resolve today. Until it does, the script above — calling the public REST API directly with curl and jq — is the real, working way to run test packs from CircleCI. It does the same three things the orb does: authenticate, trigger a run, and check the result.
  • Does this work with self-hosted Total Shift Left?
    Yes. The script calls the public /api/v1 endpoints, so any reachable Shift-Left deployment works. For self-hosted deployments behind a firewall, run the job on a CircleCI self-hosted runner inside the same network as the Total Shift Left deployment.
  • How should I store credentials in CircleCI?
    Use a CircleCI Context (Organization Settings → Contexts) to scope credentials to the workflows that need them. Reference the context in your workflow definition with the context: key. Avoid project-level environment variables for credentials — they are easier to misuse and harder to rotate.
  • How do I gate workflows on test results?
    The example script exits non-zero when the pass rate drops below your threshold, which fails the job. Combine with workflow approval gates or branch protection in your VCS to block downstream jobs (deploys, releases) on API test failures.
  • Can I run multiple test packs in parallel?
    Yes. Use CircleCI parallelism with separate test-pack IDs per shard, or define multiple jobs in the same workflow. CircleCI bills by minutes, so heavy parallelism trades cost for wall-clock time.
  • When will the orb be installable?
    The source is written, tested and public; what remains is publishing it to the CircleCI registry, which also depends on the shared runner package being published first. We are not committing to a date. This REST API pattern will keep working either way — the orb is a convenience wrapper around the same endpoints, not a replacement for them.

More CI/CD integrations

Add API testing to your CircleCI workflows

Forever-free Citizen Developer or 15-day Enterprise trial. The REST API pattern above works today — no need to wait for the orb.

Book DemoSee pricing