AI Test Case Generation Tools: Strategies You Control

This is lesson 9.5 of Shift-Left API in Depth. In lesson 9.1 you generated your first tests with the default settings; now you open the box. For the demo, use the free edition of Shift-Left API and the public sandbox store spec.

How do AI test case generation tools decide what to write?

AI test case generation tools promise a full test suite in seconds. But which tests do they actually write, and can you choose? In Shift-Left API, coverage-driven strategies decide, and you choose which ones run: switch them in the AI Generation settings, generate again, and read what each one added.

One schema, many families of tests

A coverage-driven strategy is a rule for turning a contract into tests. Take the sandbox store's schema for creating an order:

• quantity has a minimum of 1. The boundary strategy tests 1, which must pass, and 0, which must fail.
• status is one of four values. Equivalence partitioning sends a value outside the set.
• notes can be 500 characters long. Boundary again: 500, and 501.
• The user ID must be a UUID. The negative strategy sends a bad format.
• Unknown fields aren't allowed. Schema fuzz sends one anyway.
• The endpoint only takes POST. HTTP semantics tries PATCH and expects a 405.

Which part of API testing using AI tools is AI?

When you do API testing using AI tools, it helps to know which part is AI. In Shift-Left API there are two layers. First, the coverage engine plans tests from your spec, strategy by strategy, with no AI at all. Then, if you've configured an AI provider, the AI reviews the set, adds tests and improves them; some strategies also guide what it writes. No provider configured? You still get every test from the spec.

The strategies and their defaults

Nine strategies are on by default: the functional families, from positive and negative through to schema fuzz. Two are opt-in: security probes send hostile input, like SQL injection, and performance checks hold each response to a time limit. You switch those on for a test environment. A severity filter keeps only the levels you tick.

Generate with the defaults

1. Open a project built from the sandbox store's spec, with endpoints but no tests yet.
2. Click Orders in the Features table, then click Create a new order. Its Tests table is empty.
3. Click Generate tests. The coverage engine plans the tests from the spec, then the AI reviews them (when an AI provider is configured, as in the video).

Read the summary from the top. From your API spec counts the tests covering the endpoint's fields, status codes and error paths. AI review says what the AI added and improved. The last row, Not covered, lists security and performance, both off in settings. Your counts can differ from the video's, because the AI part varies between runs.

Each test has a severity and a tag naming the strategy that wrote it: positive for a valid order, negative for a malformed user ID, boundary for a quantity of 1, HTTP semantics for a PATCH that must get a 405.

Switch strategies in the AI Generation settings

Which strategies run is a project setting. Click Project Settings at the top right of the page, then the AI Generation tab, and read it from the top:

• Enable strategy-based test generation. Turn it off and you only get the basic tests: the happy paths and missing or invalid input.
• Strategies. Every test always checks the status code, the response format and the data values. Below them are the eleven strategies: auth depth, boundary, decision table, equivalence partitioning, HTTP semantics, negative, pairwise, performance, positive, schema fuzz and security probes.
• Tick Performance, latency SLA and Security probes. They're off by default because they're intrusive, so point them at a test environment, not production.
• Severity filter. Tests outside the ticked levels are dropped before they're saved. Critical plus high only is handy for a release gate.
• AI test generation. Custom instructions are added to every AI request for this project, such as "IDs are UUIDs".

Click Save settings. The new settings apply to the next Generate tests run, and existing tests aren't touched. Click Close.

Generate again and read what each strategy added

Back on Create a new order, click Generate tests again; two more strategies run. The summary reports new and updated tests: a planned test that already exists is updated in place, not duplicated. Nothing is listed as not covered any more.

To read what one strategy wrote, click its tag. Click Security, and the list holds only the security probes: SQL injection, cross-site scripting, shell metacharacters, path traversal and more, plus checks like mass assignment and oversized payloads. Each one passes when the API handles the probe with no server error and doesn't honor the hostile input. (These probes are tests in the endpoint's own suite, separate from the API security testing add-on.)

Clear the technique filter, click Search tests and type SLA: Response time within SLA is the performance strategy's test.

To give a whole feature its tests with these settings, go back to its endpoints and click Generate Tests for All. The settings belong to the project, so every endpoint gets the same strategies.

Try it yourself

Build a project from the sandbox store spec and generate tests for Create a new order with the defaults. What does the Not covered row say? Then, in Project Settings → AI Generation, tick Performance and Security probes, and save. Generate again: how many new tests, and which tags do they carry?

Recap

• AI test case generation tools are only as useful as the coverage you can control: in Shift-Left API, eleven strategies decide which families of tests are written, nine on by default and two you opt in to.
• You switch them in Project Settings → AI Generation, and generating again updates what's there instead of duplicating it.
• The spec-based tests don't need an AI provider at all; the AI review adds to them when one is configured.

AI test generation is in every edition, including the free one. Next, in lesson 9.6, you open a test in the test workbench and read and write its assertions. For a wider look at how AI-based test generation tools compare, read AI testing tools.