API Test Schedule: Run Now, Test Run Packs and Nightly Runs

This is lesson 9.10 of Shift-Left API in Depth, the product track of the learning center. In lesson 9.9, one test ran against a mock server. Now you gather tests into a Test Run Pack, run it now, and schedule it for every night. Running a test by hand is in every edition of Shift-Left API, but Test Run Packs and schedules come with Professional and Enterprise, so follow the demo on a 15-day trial. The demo uses the public sandbox store API.

How do you set up an API test schedule?

An API test schedule runs your tests on a timetable, every night for example, so a regression shows up the next morning, not at the next release. The questions are how you turn a handful of tests into something that runs on its own, and how you know it ran.

What a Test Run Pack is

A Test Run Pack is a named set of tests that run together against one environment: a smoke pack of five order tests, for example. You run the whole pack in one go, from the app, from a CI/CD pipeline, or on a schedule. Each run gives one result for the set, and the pass rate of run after run shows you the trend.

How CRON describes an API testing schedule

A schedule is often written as a CRON expression: five fields, for the minute, the hour, the day of the month, the month, and the day of the week.

• 0 6 * * 1-5 means six in the morning, Monday to Friday.
• 0 2 * * * means two in the morning, every day: a nightly run.

Every scheduled run lands in the history, so the one night that failed stands out against the passes around it.

Run now, or schedule?

Run a pack now right after a change you've just made, or to try a new pack before you trust it. Schedule it for the regression run every night, for services that change without telling you, and for a steady history you can compare. That is why API testing is important to repeat: the purpose of API testing is to tell you when something broke, and a schedule asks every night.

Build a Test Run Pack

The demo project, Store API Schedules, is built from the sandbox store's spec and has five tests: they read the product list, one product, a product that doesn't exist, a user, and an order.

1. Click Execution in the top bar. There are no test run packs yet. Click New Test Run Pack. The wizard has five steps.
2. Step one, the pack's details. Functional runs individual tests; Workflow runs whole workflows. Name it Store API nightly, and add a description of what it checks and when. Enabled means the pack runs on its schedule; you can still run it by hand either way. Click Next.
3. Step two, the tests. Open the Project list, and pick the project. Its five tests appear, each with its feature and endpoint. Click Select All: five of five selected. They run in this order, and you can drag them to change it. Click Next.
4. Step three, execution preferences. With Override off, the pack uses the project's settings, so it runs on the project's default environment: Development, the live sandbox. Click Next.
5. Step four, notifications, which can tell you how a run went. Leave them off, and click Next once more.

Schedule it for every night

Step five sets the schedule: a start date, a time of day and a time zone, then Repeat: once, every few minutes, every few hours, daily, weekly, monthly, or custom.

1. Choose Daily. Set the time of day to 2:00 a.m., and let it run forever.
2. Read the preview: it lists the next five runs, at two in the morning, every night, in the time zone shown.
3. Click Schedule Test Run. The pack is saved; close the confirmation.

The pack's row shows it as functional, five tests, idle. Upcoming Schedule shows its next run: tomorrow, at 2:00 a.m. The Engine's scheduler checks for due packs every 30 seconds, so tonight the pack runs on its own, as long as the Engine is running. The switch on the left of the row pauses the schedule, and keeps the pack.

Run it now, and read its history

You don't have to wait for tonight.

1. Click Run Now, the green play button. The pack runs, and when it finishes, its summary opens: it ran on Development, on the Engine, with five passed, none failed, five in total.
2. Click View results. This is the run's High Level Test Report, in Reporting: the pack, its project and its environment, and the execution type, Manual, because you started it. The pass rate is 100%.
3. Go back to the list of runs, and search for the pack's name. This is the pack's history: each run, with its time, its type and its results. Tonight's run will join it, as Automated.

Try it yourself

On the sandbox store, with a trial edition, import the spec from https://demo.totalshiftleft.ai/openapi/rest.json, and make a few tests that read its products, a user and an order. In Execution, click New Test Run Pack, and select those tests. On step five, set Repeat to Daily at 2:00 a.m., and schedule it. Then click Run Now, read the summary, and find the run in Reporting.

Recap

• An API test schedule runs a set of tests on a timetable, so you find out about a regression the morning after.
• In Shift-Left API, a Test Run Pack holds the tests, its fifth step sets the schedule, and Upcoming Schedule shows the next run.
• Run Now runs the pack right away, and Reporting keeps every run, by hand or on schedule: scheduled API tests and nightly test runs, without anyone at the keyboard.

The 15-day trial includes Test Run Packs and schedules, plus the Performance testing add-on and the API security testing add-on. Lesson 9.11 covers Run Triage and Fix with AI for failed runs, and lesson 9.12 turns your runs into dashboards. For where scheduled runs fit in API testing, read what API testing is and how it works.