Create and Run a Performance Test

Overview

This article walks you through designing a performance test in Shift-Left Studio, checking it with a dry run, running it, watching it live, and finding out why a run failed. It also covers scheduling, notifications, and setting up load agents for more load than one machine can send.

A performance test (called a scenario on screen) is built from tests you already have. The designer has five steps: Kind & name, Workloads & steps, Load profile, Targets & SLOs, and Target & review. You can move back and forth between them, and nothing is sent to your API until you start a run.

For what each kind of test is for, see Performance Testing: Load, Stress, Spike, Soak and More.

Key concepts

• Workload: a group of virtual users doing the same thing. One workload is the default.
• Stage: one row of the load profile, with a duration, a target, a shape and a role. The hold is judged; warm-up and ramp-down are not.
• Threshold: a pass/fail rule, such as p95 latency during the hold < 800 ms.
• Dry run: builds the run exactly as a real run would, without sending any load.
• Check before load: one request per step, sent just before load starts. If it fails, no load is sent.

Before you begin

• A Professional, Trial or Enterprise licence.
• Tests in the project that pass in a functional run against the environment you want to load.
• An environment with a base URL and working authentication. The load test uses the environment's authentication, exactly as a functional run does; there is no separate authentication setting.
• If you plan to create records under load, a data set with enough rows (see Step 3).
• Permission from whoever owns the target system to send load to it.

Step 1 — Open the designer

1. Select your project, then click Performance in the top toolbar.
2. Click New performance test.

You can also start from somewhere you already are:

• On a test's row, click Run as load test. The designer opens with that test as its only step and the smoke preset.
• On a test run pack, click Create load scenario from this pack. The pack's tests become the steps.

Step 2 — Choose the kind and name (Kind & name)

1. Under What should this test find out?, click a kind, for example Load or Stress. Each card says what it answers and whether it uses a fixed number of users or a fixed rate of requests.
2. Enter Expected users (or Expected requests per second for rate-based kinds). This is your normal busy-day figure; the preset scales from it.
3. Enter Target p95 (ms): 95 of every 100 requests should answer faster than this.
4. Enter a Name, for example "Checkout on a normal busy day", and optionally a Description (optional).

Note: Once you edit the load profile or targets by hand, these three numbers stop rewriting them. Click Reapply the preset if you want to start over. If you change the kind later, you are asked whether to Use its preset or Change only the kind.

Step 3 — Choose what each user does (Workloads & steps)

1. In Workload name, name the group of users, for example "Browse".
2. Under What this workload runs, choose Tests or A workflow.
3. If you chose Tests:
   1. Search the list with Search tests by name, endpoint, feature or tag, and click Add beside each test you want.
   2. Under How the steps are used, choose Run these in order as a user journey (for example, log in → search → add to cart) or Mix these requests by weight (each iteration picks one request, weighted by the Weight you set).
   3. Use the arrows to reorder steps and Remove to take one out.
4. If you chose A workflow, pick it from Workflow. Its calls run in order for every virtual user, with its data wiring.
5. Under Pacing, choose one:
   • Pause between steps (think time), then set Think time from (ms) and to (ms). Each pause is picked between the two.
   • Start a journey at a fixed interval, then set Interval between journeys (ms).
6. Optionally choose a Data set (optional) and How rows are used:

7. To mix traffic, click Add workload and give each workload a Share of the load (%). Shares must add up to 100%, for example 80% browse and 20% checkout.

Important: A step marked writes data creates, changes or deletes something on every iteration. Point these at a test environment, and use a data set in "each row used once" mode where duplicates would collide.

Step 4 — Shape the load (Load profile)

1. Under How the load is expressed, keep the preset's choice or switch between A fixed number of users and A fixed rate of requests.
2. Edit the stages. Each row moves the load from where the previous stage ended to its own target:

3. Click Add stage to add a row. Use the arrows to reorder and Remove to delete.
4. Check the Profile preview chart and the Total duration and Peak figures. If the profile goes over your edition's limit, a warning says so and the run will be refused until it fits.
5. Optionally open Advanced settings:

Step 5 — Set pass/fail rules (Targets & SLOs)

1. Under Thresholds, review the rules from the preset. Each row is read back as a sentence so you can check it.
2. Click Add threshold and choose a metric (p50 latency to p99.9 latency, slowest request, mean latency, error rate, check failure rate or throughput), a scope (Whole run, During the hold, or one step), a comparison and a value.
3. Tick Stop the run on a threshold if missing it should end the run early, and set after how many breached intervals in a row.
4. Under Use project SLOs, tick any project performance targets you want this test to judge. (To create them, see Read a Performance Report.)

Tip: A run passes only when every threshold holds. A scenario with no thresholds still reports its figures, but without a pass or fail.

Step 6 — Choose the target and check the plan (Target & review)

1. In Environment, choose where the load goes. Project default uses the project's default environment. A production environment is marked Production (locked) unless an Administrator has allowed load testing on it.
2. If load agents are available, choose under Where the load comes from: This server only, or Split across load agents (see Step 11).
3. Optionally tick Run on a schedule and set How often (Every hour, Every day, Every week), At, On and Time zone.
4. Optionally tick Tell people when a run finishes, tick Only when it does not pass if you want fewer messages, and enter Email recipients, a Slack webhook URL, a Teams webhook URL or an Other webhook URL.
5. Click Run a dry run. Studio saves the scenario and builds the run exactly as it would start, without sending any load. It shows Ready to run. Nothing has been sent yet. (or why it cannot run), the estimated Duration, Requests (estimate) and Peak in flight (estimate), the target base URL, which requests change data, each step with its expected status and checks, and any warnings.
6. The first time on a project, tick I am authorised to send load to these environments. This is asked once per project and recorded with your name.
7. Click Save, or Save & run to start straight away.

Important: A scheduled scenario that targets production asks you to type the environment's name when you save it, because nobody is watching when it runs.

Step 7 — Start a run

From the Performance home page, click Run beside the scenario (or use Save & run in the designer).

• If this is the first run on the project, the Confirm you may send load dialog asks you to confirm you are authorised. Tick it and click Start the run.
• If the environment is production and an Administrator has allowed load testing on it, the This is a production environment dialog asks you to type the environment's name. Type it and click Start the run.

Studio then sends one request per step as a check before load. If every step answers as expected, load starts. If not, the run is refused with the reason and no load is sent.

Step 8 — Watch the run live

The live monitor opens automatically. To return to it later, click Open live beside a running scenario on the home page. It shows:

• The current stage, elapsed time and environment.
• Throughput, Active users, Errors and Check failures, with live charts and a table per step.
• Thresholds: each target this interval, met or breached, with a countdown when a breach is about to stop the run.
• Warnings, such as The load generator itself is overloaded — latency now describes this machine, not the API.

Step 9 — Stop a run

1. Click Stop in the monitor header.
2. In Stop this run?, click Stop run (or Keep running to cancel).

What ran so far is kept and reported. A run you stop has no pass or fail verdict, because its profile did not finish.

Step 10 — Review run history and why runs failed

1. On the Performance home page, click All runs and why they failed. The same list appears in Reporting under the Performance runs tab, beside Functional & workflow runs.
2. Each row shows the scenario, when it started, its outcome, throughput, errors, and Why in one line.
3. Filter by Outcome, Cause, Environment, From / To date, or Scenario name contains. Click Clear filters to reset.
4. Above the list, Why runs failed counts each cause across the matching runs, for example "Requests timed out — 7 of 20 runs". Click a cause to Show only the runs with this cause. Runs per day shows how many runs passed or failed each day.
5. Click Open to see a run's full report, or Compare to compare it with its baseline.

Outcomes you will see:

Each run's report explains its causes in detail. See Read a Performance Report.

Step 11 — Set up load agents (optional, Administrators)

Load agents let a run send more load than one machine can, or send it from several places. They need a Trial or Enterprise licence.

1. On the Performance home page, click Load agents, then Register a load agent.
2. Enter a Name people will recognize, such as the machine it runs on, and optionally Most users it should carry (a guide for test designers). Click Register.
3. Studio shows the agent's secret once, with the exact settings to start it. Use Copy the environment variables (recommended: a command line is visible to every user of the machine) or Copy the start command. Click I have copied it.
4. Install the Studio local runner on the agent machine, set the copied values, and start it as a load agent (api-testing-local-runner --role=load-agent).
5. The agent connects out to Studio; nothing needs to be opened on its firewall. When it connects, its status shows Online.

To use agents in a scenario, go to Target & review, choose Split across load agents, tick the agents, and decide whether This server sends a share too. Each generator sends an equal share of the users or rate, with its own share of the data rows, and they all start at the same moment.

Replace secret makes the old secret stop working at once. Revoke refuses the agent on its next call and stops any load it is sending.

Note: A scenario that authenticates with a client certificate always runs from the server only. The private key never leaves the server.

Troubleshooting

Best practices

• Run a Smoke test first, then Load, before trying Stress or Breakpoint.
• Run a dry run after every change; it shows exactly what will be sent.
• Keep Stop automatically when most requests fail on unless you are deliberately probing failure.
• Use realistic think time; zero think time produces far more load per user than real people do.

FAQ

Why does a scheduled run sometimes not start? If Studio was down for more than 15 minutes at the scheduled time, the missed run is skipped and recorded rather than started late.

Can two people run the same scenario at once? No. A scenario runs once at a time; start it again when the current run finishes.

Automating this

With an API key you can start a scenario from CI:

1. POST /api/performance/scenarios/{scenarioId}/run starts the run and returns its id.
2. GET /api/performance/runs/{runId} returns its status; repeat until it finishes.
3. GET /api/performance/runs/{runId}/junit.xml returns a JUnit report: one test case per target plus a run-verdict case, so your pipeline fails when the run fails. The report includes why the run failed.

Related articles

• Performance Testing: Load, Stress, Spike, Soak and More
• Read a Performance Report: Percentiles, Verdicts and SLOs
• Environments
• Test Run Packs
• Data Sets