intermediate·9 min read·Updated Oct 24, 2026

What Is Test Data Generation? Methods, Examples and Where AI Fits

Test data generation creates the inputs and expected results your tests need. Here are six methods, a worked example on one endpoint, and an honest look at test data generation with AI.

This is lesson 6.3 of Test Data, Mocks and End-to-End API Testing. Lessons 6.1 and 6.2 replaced the service; this one is about the data you send to it. It helps to know negative testing first.

What is test data generation?

What is test data generation? It's the work of creating the inputs a test sends and the results it expects back, deliberately, instead of typing whatever comes to mind. For an API, that means request bodies, path and query values, headers, and the expected status and fields for each. Good generated data covers the valid cases, the edges of every rule, and the invalid cases the API must refuse, and it can be produced again tomorrow.

Six ways to generate test data

Method	How it works	Good for	Watch out for
Hand-written	A person writes each row	A few meaningful business cases	Gaps; it stops at the cases someone thought of
Schema and rule based	Read each field's type, format, enum, min and max, and derive values	Boundary and invalid inputs, at scale	Only as good as the spec
Combinatorial	Equivalence classes, boundary values, pairwise combinations	Many parameters with few values each	Row counts explode without pairwise
Random (faker-style)	Generators for names, emails, dates	Volume, uniqueness, realistic-looking values	Unrepeatable unless seeded
Production-derived	Subset real data, then mask it	Realistic distributions and odd records	Privacy, and masking that leaks
AI-generated	A model writes rows from a schema or a description	Realistic, domain-flavoured variety	Constraint errors, drift between runs

Most teams combine three: rule-based for coverage, a few hand-written business rows for meaning, and random or AI values for realism.

A test data example on one endpoint

The sandbox store's POST /api/v1/users takes name (required, 1 to 100 characters), email (required, email format), role (user, admin or moderator) and age (an integer from 0 to 150). Deriving rows from those rules gives:

#	Row	Why	Expected
1	`{ "name": "Alice Johnson", "email": "alice@demo.com" }`	Valid minimal	201, `role` defaults to `user`
2	`{ "name": "Bob", "email": "bob@demo.com", "role": "admin", "age": 34 }`	Valid full	201
3	`age: 0`	Lower boundary	201
4	`age: 150`	Upper boundary	201
5	`age: 151`	One past the maximum	400, field `age`
6	`age: -1`	One below the minimum	400, field `age`
7	`email: "notanemail"`	Bad format	400, field `email`
8	`role: "superadmin"`	Outside the enum	400, field `role`
9	`{ "email": "alice@demo.com" }`	Required field missing	400, field `name`

Nine rows from four field rules, and each one carries its own expected result. That last column is what turns a list of values into test data. Try row 5 against the live sandbox:

POST/api/v1/users

A boundary row: age 151 is one past the documented maximum of 150, so the store answers 400 with the field name.

curl -X POST 'https://demo.totalshiftleft.ai/api/v1/users' \
  -H 'Content-Type: application/json' \
  -d '{"name":"Gus Moreno","email":"gus@demo.test","role":"user","age":151}'

The store answers 400 with a validation error naming age. The sandbox's own requirements document, /requirements/store.txt, lists rows like these as TD-U-01 to TD-U-15, in the same input-and-expected format.

Test data generation with AI: what it's good at

Language models are useful in two places:

Realism. "A name, a company and a postal code for a customer in Lyon" gives values that look like real records, which surfaces bugs in validation that only show on real-looking input: accents, apostrophes, long surnames.
Prose to rows. A business rule written as a sentence, "orders over 10 items need a manager's approval", becomes a pair of rows on either side of the threshold.

Test data generation using AI: what goes wrong

Constraints are approximate. A model asked for ages from 0 to 150 will sometimes write 151. Validate every row against the schema before you keep it.
It isn't repeatable. The same prompt gives different rows tomorrow, so a failing test can't be replayed. Generate once, review, and keep the result as a file in version control.
Duplicates and blandness. Models drift towards the same few names. Check uniqueness where the API requires it.
Privacy. Pasting production records into a prompt sends them to the model provider unless the model runs in your own network. Generate from schemas and rules instead.

The rule of thumb: let AI propose values, let rules decide whether they're valid, and let a file decide what the test sends.

Common mistakes

Data without expected results. A row with no expected status can't fail a test.
Only valid data. Boundary and invalid rows find more bugs per row than happy ones.
One giant shared sheet. Tests that share rows step on each other; give each suite its own set (more in lesson 6.4).

Where test data comes from in Shift-Left API

Shift-Left API gets test data from three places, and only one of them needs an AI model:

The spec, with no AI at all. Test generation reads each field's required flag, type, format, pattern, enum, min and max, and plans boundary and negative tests from them: at and beyond the minimum and maximum, too short and too long, wrong type, invalid enum. Every generated test comes with its assertions.
Your own sheets, in Requirements → Data for Generating Tests. Upload an Excel or CSV test-data sheet and generation spreads its real rows across the positive tests. Columns named INVALID_<column> become negative inputs, and a sheet with several rows becomes data-driven variants of a test. If you've connected an AI model, generation can also replace placeholder values in positive tests with realistic ones taken from your requirements and fixtures; it adds at most 8 extra tests per endpoint and never removes or weakens a planned one.
Data sets, pushed into tests you already have. Fill tests from data sends a data set's values into a project's existing tests, with a preview first and an undo after:
1. In Project Settings → Data for Running Tests, import a sheet as a data set.
2. Click Fill tests… in the Fill tests from data panel.
3. Choose Fill values — each test gets one row, or Run each passing test once per row.
4. Click Preview. It counts the tests that will change and the fields filled, lists old and new values, and names columns no test sends.
5. Click the Fill button, which names how many tests it will change. Recent fills keeps an Undo.

A fill only writes into fields a request already sends, converts values to each field's type, and leaves deliberate boundary and negative values alone unless you untick Keep values a test chose on purpose. INVALID_ values are never written into a test that must pass, and credentials and transport headers are never filled. The fill tests from data feature page has the full rules.

Try it yourself

Import the sandbox spec and generate tests for POST /api/v1/users. Import the users sheet from lesson 9.7 as a data set, click Fill tests…, choose Fill values — each test gets one row, and read the preview before you apply it.

Recap

What is test data generation: deliberately creating inputs and their expected results, covering valid, boundary and invalid cases, in a way you can repeat.
Test data generation with AI adds realism and turns prose rules into rows; validate every row against the schema and keep the accepted set as a file.
Test data generation using AI is safe for regulated work only when you control where the prompt goes; rules and schemas are a safer source than production samples.

For generator libraries, factories and a full checklist, read how to generate test data for API testing.

Next: lesson 6.4, what is test data management.

Hands-on lab

Try it: Import https://demo.totalshiftleft.ai/openapi/rest.json, generate tests for POST /api/v1/users, import the users.csv sheet from lesson 9.7 as a data set, then click Fill tests…, choose Fill values — each test gets one row, and click Preview.

Frequently asked questions

What is a good test data example for one API endpoint?+

For a user sign-up endpoint with name, email, role and age, a useful set has one valid minimal row, one valid full row, boundary rows for age (0, 150, 151 and -1), one bad email, one role outside the allowed list and one missing name. Each row carries its expected result: 201 for the valid ones, 400 and the field name for the rest.

How does test data generation with AI work?+

You describe the data you want, or hand the model a schema and some business rules, and it writes rows: realistic names, plausible addresses, values that fit the domain. Models are good at variety and realism and weak at exact constraints and repeatability, so validate every generated row against the schema and keep the accepted set as a fixed file.

Is test data generation using AI safe for regulated data?+

Only if you control where the prompt goes. A prompt that includes real customer records sends them to the model provider unless the model runs inside your own network. Generate from schemas and rules rather than from production samples, and use a self-hosted model when the rules themselves are sensitive.

Related lessons

Test Data & Mocks

What Is Synthetic Test Data Generation, and When Is It Enough?

Synthetic test data looks like real data and contains no real people. Here's what synthetic test data generation means, the four ways to do it, where generative AI fits, and a seeded data-driven run of synthetic users.

Test Data & Mocks

What Is Test Data Management in API Test Automation?

Test data management keeps the data your automated tests rely on known, isolated and resettable. Here's why it's needed, the five strategies, and how environments, data sets and a reset call fit together.

Shift-Left API in Depth

Sample API Test Cases in Excel Sheet: Data-Driven Testing

A row per case, with its inputs and its expected result: that sheet is already a test suite. Import it into Shift-Left API as a dataset, bind its columns to a test, and run it once per row.