API Test Automation Framework: GitHub Actions, Azure DevOps and Jenkins

This is lesson 9.15 of Shift-Left API in Depth, the product track of the learning center. Lesson 9.14's security checks, like every test in a pack, can run from CI. For the demo you need a trial edition of Shift-Left API, because API keys and the public API a pipeline uses aren't in the Free edition, and the public sandbox store API.

How do you run an API test automation framework in your CI pipeline?

An API test automation framework earns its keep in the pipeline: every build runs the API tests, and a build that breaks them stops. A pipeline reads one thing from a step: its exit code. Zero, and the build goes on. Anything else, and it stops. So wiring it up means one step that runs your tests and sets that exit code.

API tests in the pipeline

Shift-Left API keeps the tests; your CI decides when they run. On every push, one step runs the API tests between the build and the deploy, and a quality gate passes or fails the build.

There are two ways in:

• Ready-made integrations, published on GitHub: a GitHub Action, an Azure DevOps task and a Jenkins plugin. They sign in as an API user and gate on a pass threshold in percent.
• The headless runner: one node command with an API key and a pack ID. It isn't in the installer yet. The lesson runs it by hand, to show what every CI step does.

The quality gate

The gate is one rule. With a minimum pass rate of 0.9:

• Six of six passed: exit zero.
• Nine of ten is exactly 0.9: still exit zero.
• Five of six is 0.83: exit one, and the build fails.

Without a minimum, any failed test fails the build. And if the server can't be reached, or the pack never finishes, the exit code says so: never a pass.

The pack a pipeline runs

The demo project, Store API CI, is built from the sandbox store's spec, with six read tests.

1. Click Execution in the top bar and search for its pack, Store API regression: six tests, and no schedule, because a pipeline will run it.
2. A pipeline names a pack by its ID. Hover the ID icon to see it, and click it to copy it.

Create a scoped API key

The runner signs in with an API key.

1. Open the account menu and click Settings.
2. Open Access and Identity, then API Keys.
3. Name the key, for example Store API pipeline, and set when it expires. The demo uses one day.
4. Tick the scopes. A key can do only what you tick, and never more than the person who made it. Under Test Execution, tick Execute tests manually; under Reporting, tick View test execution reports.
5. Click Create API Key. The secret is shown once: copy it into your CI's secret store.

The step in GitHub Actions, Azure DevOps and Jenkins

Written as a pipeline step, the runner takes the server's address, the key, the pack's ID, a JUnit file and a minimum pass rate of 0.9.

• GitHub Actions: the key comes from a repository secret, and a JUnit report step then shows each test, even when the gate failed.
• Azure DevOps: the same command in a script step, with the key as a secret variable, then Publish Test Results.
• Jenkins: the same command in an sh step, with the key from a Jenkins credential, and junit to record the results.

With a ready-made integration, the plugin is that step: the same server, pack and threshold, signed in as an API user.

Pass, then fail the build

In the video, the step runs by hand on the presenter's PC with the new key. The runner starts the pack and waits for it. It writes results.xml, six of six pass, and the exit code is zero, so the build goes on.

Then a change ships to the store: the Wireless Mouse's price goes from 24.99 to 29.99. Run the step again, and five of six pass. That's below 0.9, so the exit code is one, and the pipeline fails the build.

Find the pipeline's runs in Reporting

Every run the pipeline starts lands in Reporting, like any other. Search for the pack: the newest run shows five passed and one failed, the one before it all six. Open the newest to see the pass rate, eighty-three percent, and the test that broke the build: Read the Wireless Mouse, which checks the price.

Try it yourself

With a trial and the sandbox, copy your pack's ID from Execution. Keep an API user's email and password in your CI's secrets. Add the ready-made GitHub Action, Azure DevOps task or Jenkins plugin from GitHub, on a CI agent that can reach your server, with a pass threshold of ninety percent. Then break a test on purpose, push, and watch the build fail.

Recap

• An API test automation framework belongs in the pipeline: one step runs a Test Run Pack, writes JUnit, and sets the exit code from a pass-rate gate.
• In GitHub Actions, Azure DevOps and Jenkins, a ready-made integration is that step, signed in as an API user.
• As an automated API testing framework it needs no code, because the tests live in Shift-Left API, and every pipeline run lands in Reporting.

Next, in lesson 9.16, you bring your own LLM to Shift-Left API, from a hosted provider or one you run yourself with Ollama, vLLM or LM Studio. The whole track is on the Shift-Left API in Depth page. For how to weigh framework options, read how to choose an API testing framework.