API Testing With Jenkins: Jobs, Gates and JUnit Results
Jenkins runs API tests like any build step: an exit code, a JUnit report and a build result. See how to structure the job, gate on FAILURE or UNSTABLE, and run a Test Run Pack with the Shift-Left API plugin.
This is lesson 7.3 of API Testing in CI/CD, Regression and Performance. Lesson 7.2 ran a test pack from GitHub Actions; this lesson does the same job in Jenkins, which many regulated teams still run on their own servers.
How does API testing with Jenkins work?
API testing with Jenkins works the way every Jenkins build step works: a step runs your API tests, its exit code decides whether the build fails, and a JUnit file tells Jenkins which tests passed. Jenkins adds two things on top: a build result with more than two states, and a test trend across builds. The decisions you make are where the job runs, how the failure maps to a build result, and how the results get reported.
Freestyle jobs and Pipelines
Jenkins has two job styles, and both run API tests well:
| Freestyle job | Pipeline (Jenkinsfile) | |
|---|---|---|
| Defined in | The web UI, stored by Jenkins | A Jenkinsfile in your repository |
| Steps | Build steps and post-build actions, picked from lists | stage and steps blocks in code |
| Good for | A single test job, plugin build steps with a form | Multi-stage build, test and deploy flows, reviewed like code |
| Reports | Publish JUnit test result report post-build action | The junit step |
A common shape uses both: the Pipeline builds and deploys, then calls a separate test job with build job: 'store-api-tests'. By default the build step waits for that job, and a failed or unstable downstream build carries its result back to the Pipeline.
Where the API tests go in a Jenkins pipeline
The order is the same as any pipeline from lesson 7.1: build, deploy to a test environment, run the API tests against it, then promote. Three Jenkins specifics:
- Agent placement. The agent that runs the step must reach whatever runs the tests and, if the tests run locally, the API under test. Use agent labels to pin the job to an agent inside the right network.
- Credentials. Store logins as Jenkins credentials (type Username with password) and bind them into the step; never type them into a job's configuration or a
Jenkinsfile. - Concurrency. Two builds of the same job testing one shared environment collide. In a Pipeline,
options { disableConcurrentBuilds() }stops that; in a Freestyle job, leave Execute concurrent builds if necessary unticked.
Gates: FAILURE, UNSTABLE and SUCCESS
A Jenkins build ends in one of several results, and three matter for API tests:
| Result | Colour | Use it when |
|---|---|---|
| SUCCESS | Blue or green | The gate passed |
| UNSTABLE | Yellow | Tests failed, but you want the build to finish and report, not block |
| FAILURE | Red | The gate failed and nothing downstream should run |
The JUnit plugin marks a build UNSTABLE when the report it reads contains failures, even if every step exited zero. So decide on purpose: either the test step itself fails the build at the gate, or you let the JUnit report set UNSTABLE and add a rule downstream that refuses to deploy an unstable build. Mixing the two by accident is how a red API suite ends up deployed.
A worked example against the sandbox store at https://demo.totalshiftleft.ai: a pack of eight read tests with a 90 percent gate. Eight of eight is SUCCESS. Seven of eight is 87.5 percent, under 90, so the gate fails: FAILURE if the guard is strict, UNSTABLE if you chose to warn. The JUnit report shows the one failing test by name either way, and its trend graph shows when it started failing.
Does Jenkins have an API?
Yes, and people often mean this when they search for Jenkins and APIs together. Jenkins exposes a remote access API: append /api/json to most URLs, such as a job or its last build, to read the same data the page shows. Another system can start a job by POSTing to its /build or /buildWithParameters URL, authenticated with a user's API token. Release tools use it to start a test job and read its result.
That API manages Jenkins. It doesn't test anything. To test your own APIs, the job runs a test tool, which is the rest of this lesson.
In Shift-Left API: run a Test Run Pack from Jenkins
Shift-Left API ships a Jenkins plugin, the Shift-Left API Automation Integration, which adds a build step that runs a Test Run Pack and applies a pass-rate gate. Test Run Packs and CI need the Professional or Enterprise edition, or the 15-day trial.
The plugin isn't on the Jenkins Update Center, so you install it by hand from its .hpi file:
- Download the
.hpifrom the GitHub releases ofTotal-Shift-Left/Shift-Left-API-Integrations. - In Jenkins, open Manage Jenkins → Plugins, then the Advanced settings tab.
- Under Deploy Plugin, choose the file, click Deploy, and restart Jenkins when it asks.
Then configure the job:
- Credentials. Add a Jenkins credential of type Username with password: the username is the email of a Shift-Left API user whose role is allowed under Settings → Integrations → Public API, and the password is that user's password.
- Build step. In a Freestyle job, click Add build step and choose Shift-Left API Automation Integration: Run Test Pack.
- Connection. Enter the Server URL of your Shift-Left API server, with no trailing slash, pick the credential, and click Test Connection. It answers Login OK.
- Test Pack. Pick your sandbox pack from the Pack ID list, which the plugin fills from your server.
- Quality Gate. Set Pass threshold (%) to 90. Leave Fail build when any ERROR tests ticked. Under Build result on gate failure, choose FAILURE to block, or UNSTABLE to warn.
- Artifacts. Tick Write JUnit XML; it is off by default. Then add the post-build action Publish JUnit test result report with the file
shiftleft-test-pack-results.xml. - Click Build Now.
The visible result is in the console output: the plugin signs in, notes the pack's previous run, starts the pack, polls every 10 seconds until a new run finishes, and prints [ShiftLeft] Decision: PASSED (result=SUCCESS). The build page gets a Shift-Left API Automation Integration link with the run's summary, and the JUnit report lists every test. Break one test in the pack and run it again: below 90 percent the decision becomes GATE_FAIL_THRESHOLD, and the build takes the result you chose.
A few facts to plan around:
- Freestyle today. The plugin is a Freestyle build step. From a Jenkinsfile, call the Freestyle job with
build job:, or call the public REST API under/api/v1from anshstep. - Where the tests run. The pack runs on the Shift-Left API server, not on the Jenkins agent. The agent only needs to reach the server; the server must reach the API under test.
- One environment per pack. A job that tests dev and staging runs two packs, one per environment.
- No double runs. If the pack is already running, from a schedule or another build, the start is refused rather than queued.
The Jenkins integration page lists every field. Lesson 9.15 shows the same gate across GitHub Actions, Azure DevOps and Jenkins in one demo.
Common mistakes
- Forgetting the JUnit step. The build fails, but nobody can see which test without reading the console.
- Running on the controller. Run test jobs on agents; the controller should schedule work, not reach into test networks.
- An UNSTABLE build that still deploys. If you warn rather than fail, make sure the deploy step checks for it.
- Credentials pasted into a job. Use a credential, so they are masked in logs and can be rotated in one place.
Try it yourself
On a trial, install the plugin's .hpi through Manage Jenkins → Plugins → Advanced settings. Add the Run Test Pack build step to a Freestyle job with a Username with password credential, click Test Connection, pick your sandbox pack, tick Write JUnit XML and build. Look for Login OK and then [ShiftLeft] Decision: PASSED (result=SUCCESS) in the console.
Recap
- API testing with Jenkins is a build step with an exit code, a JUnit report and a build result you choose: FAILURE to block, UNSTABLE to warn.
- Jenkins for API testing needs an agent that can reach your systems, credentials stored as Jenkins credentials, and no concurrent builds against one environment.
- Does Jenkins have an API? Yes, for managing Jenkins; to test your own APIs, run a test step such as the Shift-Left API plugin's Run Test Pack, installed from its
.hpi.
For Jenkinsfile patterns, shared libraries and parallel stages, read how to run API tests in Jenkins pipelines. Next, in lesson 7.4, you run the same pack from Azure Pipelines.
Try it: With a trial, install the plugin's .hpi through Manage Jenkins, add its Run Test Pack build step to a Freestyle job with a Username with password credential, click Test Connection, pick your sandbox pack, tick Write JUnit XML and build.
Frequently asked questions
Is Jenkins good for API testing?+
Jenkins for API testing works well because API tests are just another build step: they run, return an exit code, and write a JUnit file that the JUnit plugin turns into a test report and a trend. The parts to plan are where the agent runs, so it can reach the API, and how a failed gate maps to a build result.
Does Jenkins have an API?+
Yes. Most Jenkins pages have a machine-readable twin: add /api/json to a job or build URL to read its status, and POST to a job's /build or /buildWithParameters URL with a user's API token to start it from another system. That API manages Jenkins itself; it isn't a tool for testing your own APIs.
In API testing with Jenkins, should a failed gate fail the build or mark it unstable?+
Fail it when the tests guard a release, so nothing downstream runs on a broken API. Mark it UNSTABLE while a new suite is being tuned, or for a nightly report where you want the build to finish and the failures to show in yellow rather than block anything.
Related lessons
A hands-on lab: write the workflow file, store the secrets, pick a runner that can reach your test server, gate on a pass rate, keep the JUnit file, then break a test on purpose and watch the job fail.
Azure DevOps holds API test cases in two places: Test Plans, where people write and track them, and Pipelines, where automation runs them. See how the two connect, and run a Test Run Pack from an Azure Pipelines task.
Every build runs the API tests, and a build that breaks them stops. See what a pipeline step does with a Test Run Pack, create a scoped API key, gate on a pass rate with JUnit results, and find the runs in Reporting.