API Performance Testing Tools: Load, Stress, Spike and Soak Tests

This is lesson 9.13 of Shift-Left API in Depth, the product track of the learning center. In lesson 9.12, a dashboard showed every run of your tests, one request at a time. Now you send many at once. For the demo you need a trial edition of Shift-Left API, because the Performance testing add-on isn't in Free, and the public sandbox store API.

How do you load test an API with the tests you already have?

API performance testing tools answer the question your functional tests can't: does your API stay right, and fast, when many people call it at once? The second question is whether you can find out without writing a separate load script. In Shift-Left API you can: a performance test is built from your existing tests.

Your functional tests, under load

A performance test reuses your functional tests. Picture a checkout built from a REST, a SOAP and a GraphQL test. You set the traffic and the targets. A live monitor shows the throughput, the p95 latency and the errors, and the verdict is judged on the steady part of the run: the hold.

Load, stress, spike and soak

Each kind of test asks its own question:

• Load: does it hold at the traffic you expect?
• Stress: climbing in steps to twice that, where do the targets break?
• Spike: does it survive a sudden burst, and recover?
• Soak: after an hour or more of normal traffic, is it slowly getting worse?

Each counts its load in one of two ways. Load and soak tests use a fixed number of users, each one waiting for an answer, as people do. Stress and spike tests use a fixed rate of requests, which keeps arriving even when the API slows down. With a fixed number of users, a slow API would quietly get fewer requests, and hide the problem.

The Performance home

The demo project, Store API Performance, is built from the sandbox store's spec, with three read tests. Click Performance in the top bar. The home lists the project's performance tests: here a short soak, a spike and a stress test, made earlier from the same tests. To make one, click New performance test.

Design a load test in five steps

1. Kind and name. The designer asks what the test should find out; each card is a kind, with its question. Pick Load. Set Expected users to five, gentle for a shared public sandbox, keep Target p95 at 800 milliseconds, name it, and click Next.
2. Workloads and steps. This is what each virtual user does. The list holds the project's tests: click Add beside List products and beside Read the Wireless Mouse. They run in order, as one user's journey, with a pause between steps. The load test is built from the tests you already have, with their requests, checks and environment.
3. Load profile. The Load preset ramps up to five users, holds, then ramps down. Set the ramp-up to twenty seconds and the hold to sixty. On a system you own you'd hold for several minutes. The preview shows the shape, and only the hold is judged.
4. Targets. The preset's two thresholds are read back as sentences: the p95 latency during the hold under 800 milliseconds, and an error rate during the hold of one percent or less. The run passes only if both hold.
5. Target and review. The load goes to the default environment, Development, which is the sandbox, marked not production. Production stays locked unless an administrator allows it.

Click Run a dry run. It builds the run exactly as it would start and sends nothing: in the video, one minute fifty and about three hundred and seventy requests, with each step and its check. Then click Save and run.

Run it and read the verdict

The live monitor opens: throughput, active users, p95 and errors, as the run goes. The thresholds show each target, marked not judged yet during the ramp-up. A safety stop ends the run if most requests fail. When the run ends, the report takes its place. In the video it reads Pass: both targets met during the hold, at about five requests a second, with no errors and a p95 of about forty milliseconds. Below the verdict are the figures behind it.

Read stress, spike and soak reports

Click Performance to go back.

• Stress. Open the stress run, then Capacity and analysis. The test climbed in steps, from two to eight requests a second, and the chart plots the p95 and the errors against the rate reached. At this gentle rate nothing failed; on an environment you own, push higher and it shows where the targets break.
• Spike. Its analysis reads the p95 before the spike, the worst p95 during it, and the worst error rate. Then the recovery: how long the API took to get back within ten percent of its p95 from before the spike.
• Soak. A soak measures drift: how fast the p95, errors and throughput change per hour of hold. A real soak holds for an hour or more. The demo's soak held for three minutes to keep the public sandbox free, so its drift figures and verdict are noise: the fit, R squared, is near zero. Run soaks for hours, on an environment you own.

Try it yourself

With a trial edition, import the sandbox spec and add a few read tests. In Performance, click New performance test and pick Load. Add your tests as steps, with a few users and a short hold, because the sandbox is shared. Run a dry run, then Save and run, and read the verdict.

Recap

• API performance testing tools send many requests at once, and judge the answers against your targets.
• In Shift-Left API, the Performance testing add-on builds load, stress, spike and soak tests from the tests you already have.
• A dry run shows what will be sent, and each report leads with a verdict, then the analysis its kind exists for: API load testing tools and API stress testing tools in one, with no separate script.

Next, in lesson 9.14, you run OWASP checks directly against your endpoints and see the compliance mapping. To compare approaches, read API load testing tools.