LLM Testing: How to Test LLM Applications in Production
LLM testing splits the application in two. The deterministic shell — retrieval, prompt assembly, tool calls, guardrails, fallbacks — gets ordinary tests with hard assertions. The model's output gets invariant tests (it parses, it is grounded, forbidden fields are absent) plus a golden-set eval gated on mean and tail thresholds. Never assert exact wording or gate on one sample.
Reviewed by Rishi Gaurav
The most common way LLM features ship untested is not carelessness. The team tried assertEqual(output, expected), watched it fail on a re-run, and concluded the thing is untestable. It is not. LLM testing just needs the system split in two before you write a single assertion: a deterministic shell that gets ordinary tests, and a probabilistic core that gets invariants and a scored evaluation.
This guide is the LLM half of our AI API testing guide, which covers how AI changes the testing of APIs themselves. Here the subject is the reverse: testing an application that has a model in its request path — a chat assistant, a RAG search, an extraction pipeline, an agent calling tools.
In this guide
- An LLM testing strategy: split the system first
- Test the deterministic shell with hard assertions
- Handle non-determinism with invariants and tolerance
- LLM testing and evaluation: golden sets, metrics and thresholds
- Safety, security and bias testing
- Running LLM tests in CI and production
- LLM testing tools and frameworks
- LLM testing FAQ
- Sources and further reading
An LLM testing strategy: split the system first
| Component | Deterministic? | How to test it |
|---|---|---|
| Retrieval / RAG index | Yes | Ordinary assertions — which documents, whose tenant, what order |
| Prompt assembly | Yes | Snapshot the rendered prompt; assert the template and variables |
| Tool / function calls | Yes | Contract tests on the tool, plus authorization tests |
| Guardrails and filters | Yes | Hard assertions — they are controls, not preferences |
| Fallback and timeout paths | Yes | Force the failure, assert the degraded response exactly |
| Output parsing / schema | Yes | Assert it always validates |
| Generated text quality | No | Eval suite with a threshold |
| Reasoning / tool selection | No | Invariants plus eval, never equality |
Most of the table is deterministic. Teams that skip straight to "how do I evaluate quality?" leave the majority of their failure surface untested. Tool calls in particular are ordinary API surface — contract testing and authentication and authorization testing apply unchanged.
The five layers of the strategy
Once the split is clear, the strategy falls into layers that differ by what they test, how they assert and how often they run:
| Layer | What it covers | Assertion style | When it runs |
|---|---|---|---|
| 1. Component tests | Pre-processing, post-processing, parsing, caching, error handling | Hard assertions | Every commit |
| 2. Prompt and model unit tests | One prompt or model call against a small fixture set | Invariants, sampled several times | Every prompt or model change |
| 3. Pipeline integration tests | Retrieve → generate → post-process → respond, through the real API | Hard assertions on the shell, invariants on the output | Every pull request |
| 4. Quality benchmark (eval) | The full golden set, scored | Aggregate thresholds (mean, tail, per category) | Prompt, model and retrieval changes; on a schedule |
| 5. Production monitoring | Live traffic, user feedback, sampled human review | Drift alerts | Continuously |
Two things make this layering work. First, each layer is cheaper and faster than the one below it, so the cheap checks run constantly and the expensive ones run when something that affects behaviour changed. Second, findings flow upward: a production failure becomes a golden-set case, and a golden-set case that keeps failing on one property becomes an invariant test.
Why a model in the request path changes the rules
Three properties of LLM features break the assumptions ordinary test suites rely on:
- The same input does not produce the same output. Two calls can return different wording that is equally correct. Equality assertions fail constantly, engineers learn to ignore them, and the suite loses its signal.
- Behaviour changes without a commit. A provider model update, a re-indexed vector store or a new few-shot example changes what users see with no code diff. The strategy has to trigger on those changes, not only on code changes.
- Failures are quiet. A traditional bug throws an exception. An LLM failure is a summary that omits the one clause that mattered, or a confident answer drawn from the wrong document. Nothing alerts; quality has to be measured.
None of this makes the feature untestable. It means swapping exact assertions for properties, single cases for datasets, and binary results for distributions — while keeping exact assertions everywhere the system is still deterministic.
Test the deterministic shell with hard assertions
# tests/test_assistant_shell.py
def test_retrieval_never_crosses_the_tenant_boundary(api, tenant_a_token):
"""The RAG layer is deterministic and this is a security control."""
r = api.post("/v1/assistant/query", json={"q": "revenue last quarter"},
headers={"Authorization": f"Bearer {tenant_a_token}"})
assert r.status_code == 200
for citation in r.json()["citations"]:
assert citation["tenantId"] == "tenant-a", "cross-tenant document retrieved"
def test_prompt_template_is_stable(snapshot):
"""A prompt change is a behaviour change — make it reviewable."""
rendered = build_prompt(question="What is our refund policy?",
documents=[FIXTURE_DOC_1, FIXTURE_DOC_2])
snapshot.assert_match(rendered, "assistant_prompt.txt")
def test_model_timeout_degrades_instead_of_failing(api, slow_model):
with slow_model(delay_seconds=30):
r = api.post("/v1/assistant/query", json={"q": "hello"})
assert r.status_code == 200
body = r.json()
assert body["mode"] == "fallback"
assert body["answer"], "fallback returned nothing"
def test_provider_rate_limit_is_handled(api, model_returns_429):
with model_returns_429():
r = api.post("/v1/assistant/query", json={"q": "hello"})
assert r.status_code in (200, 503)
assert "Retry-After" in r.headers or r.json()["mode"] == "fallback"
def test_guardrail_blocks_the_category_it_is_supposed_to(api):
r = api.post("/v1/assistant/query", json={"q": DISALLOWED_REQUEST})
assert r.json()["refused"] is True
assert r.json().get("answer") in (None, ""), "refused but answered anyway"
Snapshot-testing the rendered prompt is the highest-value item in this list and the most often skipped. Prompts are code; a one-word change can move behaviour, and without a snapshot it lands with no diff for a reviewer to read. Version prompts in the repository, review them like code, and treat a changed snapshot as a reason to run the eval suite.
The failure paths matter as much as the happy path. Providers rate-limit, time out and occasionally return malformed output. What your application does then is fully deterministic, it is the path users hit during an outage, and it is usually the least-covered part of the system.
Streaming responses and tool servers
Assistant endpoints usually stream. If the response arrives as Server-Sent Events, the shell assertions above have to run against the assembled events, not a single JSON body — check that the stream terminates, that the final event carries the citations, and that a refusal is signalled the same way every time.
Tools exposed through an MCP server can be tested directly rather than only through the assistant: list the tools, call each with valid and invalid arguments, check the error shape, and check that a tool refuses a caller who lacks permission. How to test MCP servers covers the handshake, tool-schema and injection cases.
This is the layer where an API testing tool earns its place, and it is where Shift-Left API fits — it tests the API and MCP surface of an LLM application, not the model's answers. It parses SSE responses into individual events you can assert on, and its JSON-RPC 2.0 and MCP server testing discovers an MCP server's tools and turns each into a testable endpoint with ordinary assertions (Trial and Enterprise plans). The cross-tenant retrieval test above is a case of broken object-level authorization (BOLA), and its API security testing add-on (paid; included in the Trial) checks that systematically with a second test user and an access matrix — Deny, Own data only, Allow — across the endpoints it covers. It is not a penetration test and it is not an LLM eval framework; scored quality still belongs in the tools described below.
Handle non-determinism with invariants and tolerance
Invariants are properties that must hold for every output, regardless of wording:
# tests/test_llm_invariants.py
import json, jsonschema, pytest
SCHEMA = json.load(open("schemas/extraction.json"))
@pytest.mark.parametrize("sample", range(5)) # same input, five samples
def test_extraction_invariants(llm, fixture_invoice, sample):
out = llm.extract(fixture_invoice)
# 1. structural — it parses and validates, every time
jsonschema.validate(out, SCHEMA)
# 2. grounded — no value that is not present in the source document
assert out["total"] in fixture_invoice.text, "hallucinated total"
assert out["vendor"].lower() in fixture_invoice.text.lower(), "hallucinated vendor"
# 3. bounded — plausible, not merely well-formed
assert 0 < float(out["total"]) < 1_000_000
# 4. safe — the fields it was told to redact are absent
assert not ({"ssn", "card_number"} & set(out))
def test_refusal_is_stable_across_samples(llm):
"""Behaviour that must be deterministic gets tested as deterministic."""
outs = [llm.answer(PROMPT_INJECTION_DOC) for _ in range(5)]
assert all(o["refused"] for o in outs), "refusal is not reliable"
Sampling several times and asserting the property holds every time is what converts a flaky test into a meaningful one. If the property only holds four times in five, you have found a real defect, not a testing problem.
Four assertion patterns, in order of preference
Almost every useful assertion on model output falls into one of four patterns. Prefer them in this order, because each one down the list is more expensive and less deterministic than the one above:
- Structural. The output parses and validates against a schema or a Pydantic model. It ignores content entirely and catches the format breakage that crashes downstream code. Always the floor of the suite.
- Property-based. The output satisfies a stated property: it cites the document the question was about, a summary stays under a length cap, a translation preserves the named entities, no PII pattern appears.
- Constrained vocabulary. For classification, routing or scoring, the output must come from a fixed set. This shrinks a non-deterministic system to a small choice space and gives you a genuinely deterministic gate.
- Judged. A second model scores the output against a rubric. Useful for free-form quality that no rule captures, but the judge is itself non-deterministic and must be calibrated, so use it last and sparingly.
Ready to shift left with your API testing?
Try our no-code API test automation platform free. Generate tests from OpenAPI, run in CI/CD, and scale quality.
Tolerance and semantic assertions
When the content matters but the wording does not, assert with a tolerance rather than an exact match:
# tests/test_llm_tolerance.py
ALLOWED_INTENTS = {"billing", "shipping", "returns", "account", "other"}
@pytest.mark.parametrize("case", ROUTING_CASES)
def test_router_only_emits_known_intents(llm, case):
intent = llm.route(case.message).strip().lower()
assert intent in ALLOWED_INTENTS, f"unknown intent {intent!r}"
assert intent == case.expected_intent # deterministic once constrained
def test_answer_stays_close_to_the_reference(llm, embed):
answer = llm.answer("How long do refunds take?")
sim = cosine(embed(answer), embed(REFERENCE_REFUND_ANSWER))
assert sim >= 0.80, f"answer drifted from reference (cosine {sim:.2f})"
assert "14 days" in answer # the fact that must survive
Semantic similarity is a useful tolerance check but a weak correctness check. Embeddings of "refunds take 14 days" and "refunds do not take 14 days" sit close together, so always pair a similarity threshold with an assertion on the fact that must be present. Calibrate the threshold on real pairs you have labelled as acceptable and unacceptable; a number picked by intuition will either pass everything or nothing. Overlap metrics such as BLEU and ROUGE have the same weakness and reward copying the reference's wording, so treat them as cheap signals, not verdicts.
Remove the variance you do not need
Assert differently on the variance you cannot remove, but first remove what you can:
- Pin the model version explicitly. A provider alias like "latest" silently changes what you are testing.
- Set temperature to zero for anything asserted deterministically, and supply a seed where the provider supports one. Temperature zero reduces variance; it does not eliminate it, because outputs can still differ across calls, model versions and serving hardware. Design assertions that survive that.
- Do not force determinism the feature does not want. A creative or conversational feature that needs non-zero temperature should be tested at that temperature, with properties and evals, not tested in a configuration users never see.
- Re-run before believing a single scored failure. A single-run miss at the threshold boundary is usually variance. Gate on the aggregate and investigate the cases that fail consistently.
The stable version of a typical assertion looks like this:
# brittle — one rephrasing away from failing
assert response == "Your order #123 shipped on 4 March."
# stable — the properties that actually matter
assert response_json["order_id"] == "123" # extracted correctly
assert "4 March" in response or "2026-03-04" in response
assert len(response) < 500 # stayed concise
assert not contains_pii(response) # never leaks
LLM testing and evaluation: golden sets, metrics and thresholds
Evals answer a different question — not "is this valid?" but "is this good, and is it as good as last week?" Building an eval suite that catches regressions goes deeper on dataset and scoring design; this section is the minimum that makes an eval a release gate.
Build the golden set
A golden set is a curated, version-controlled collection of representative inputs with documented expected output properties. A working one has:
- Coverage of the real input distribution, not only the easy cases. Every category of request the feature handles in production appears.
- Hard cases on purpose: ambiguous questions, inputs in other languages, very long inputs, inputs containing your retrieval delimiter, known past failures.
- Expected properties, not expected text. "Cites document D-114, states the 14-day window, does not offer a discount" — not a paragraph to string-match.
- Provenance per entry: who added it, when, and why (a production incident number is the best "why").
- Review on change. Adding or editing an entry is a pull request, reviewed like code, not an ad-hoc append that quietly makes the suite easier.
Feed it from four sources: expert-written cases for core behaviour, adversarial cases for safety, paired cases that differ only by a demographic attribute for bias, and a regular sample of production traffic with human-rated quality to check the set still looks like reality. For a focused feature, 50 to 200 cases is usually enough for a single bad sample not to move the aggregate; weight it toward failures you have actually seen.
Choose metrics that match the feature
| Metric | What it measures | Typical method |
|---|---|---|
| Groundedness / faithfulness | Every claim is supported by the retrieved context | Claim extraction plus entailment check, or a judged rubric |
| Answer relevance | The output addresses the question asked | Judged rubric, embedding similarity |
| Context precision and recall | Retrieval returned the right documents, ranked well | Labelled relevant documents per case |
| Key-fact recall | The facts that must be present are present | Deterministic string or field checks |
| Refusal correctness | Refuses what it should, answers what it should | Labelled should-refuse cases |
| Format validity | Output parses and validates | Schema validation |
| Cost and latency | Tokens per request, model calls per flow, p95 time | Usage metadata from the provider |
Retrieval metrics deserve special attention. Most of what users experience as "the AI got it wrong" is the model generating reasonably from the wrong context. A useful diagnostic is to replay the exact retrieved context through the model offline: if the output is good with the right context and bad in production, the defect is in retrieval, and no amount of prompt tuning will fix it.
Score and gate
# evals/run_eval.py
import statistics, json
from collections import defaultdict
def score(actual, expected):
"""Combine cheap deterministic signals with a judged one."""
s = 0.0
s += 0.4 * (1.0 if expected["answer_key"].lower() in actual["answer"].lower() else 0.0)
s += 0.3 * (1.0 if set(expected["must_cite"]) <= set(actual["citation_ids"]) else 0.0)
s += 0.3 * judge_relevance(actual["answer"], expected["question"]) # LLM-as-judge
return s
by_category = defaultdict(list)
for case in GOLDEN_SET:
by_category[case.category].append(score(run(case.question), case.expected))
results = [s for scores in by_category.values() for s in scores]
mean = statistics.mean(results)
p10 = statistics.quantiles(results, n=10)[0]
print(json.dumps({"n": len(results), "mean": round(mean, 3), "p10": round(p10, 3)}))
# gate on both — a good mean can hide a bad tail
assert mean >= 0.92, f"quality regression: mean {mean:.3f} < 0.92"
assert p10 >= 0.80, f"tail regression: p10 {p10:.3f} < 0.80"
# critical categories get stricter floors than best-effort ones
FLOORS = {"account_security": 0.97, "refunds": 0.93, "smalltalk": 0.75}
for category, scores in by_category.items():
floor = FLOORS.get(category, 0.85)
assert statistics.mean(scores) >= floor, f"{category} below {floor}"
Use three kinds of threshold together:
- A static threshold — the mean and a tail metric (p10 here) must clear a fixed bar. A mean of 0.93 with a p10 of 0.4 is a feature that fails badly for one user in ten, so never gate on the mean alone.
- A drift threshold — the score may not drop more than a set margin below the baseline on main, even while it is still above the static bar. This catches the regression that the static gate lets through.
- Per-category floors — critical categories (security, money, health) get stricter floors than best-effort ones, so a strong average cannot hide a weak category.
Calibrate the judge
If you score with another model, the judge is a component that can be wrong. Before trusting it, have humans label a sample of a few dozen outputs and check that the judge agrees with them; repeat whenever you change the judge model or rubric. Judges are reliable enough to compare two versions of your feature and unreliable as an absolute score, so use them for regression detection and never as the only signal on a safety-critical property. Scoring with judges from more than one provider reduces the risk that one vendor's model shares your model's blind spots.
Safety, security and bias testing
Safety properties are controls, so most of them get hard assertions rather than scores:
- Prompt injection through retrieved content, not only through the user's message. That is the realistic attack path for RAG, and it is the first entry in the OWASP Top 10 for LLM Applications.
- Jailbreaks — inputs that try to bypass the system's constraints. Keep a versioned corpus and grow it as new techniques appear.
- Data exfiltration — attempts to extract the system prompt, other users' documents or secrets from tool configuration.
- Excessive agency — an injected instruction must never trigger a tool call, and a tool must never act beyond the caller's own permissions.
- Resource exhaustion — inputs that produce enormous outputs or long tool-call chains.
- Citation integrity — if the UI shows sources, every cited id exists and the cited document actually contains the claim.
def test_injection_via_retrieved_document_is_ignored(api, poisoned_doc):
r = api.post("/v1/assistant/query", json={"q": "summarise the report"})
body = r.json()
assert "IGNORE PREVIOUS INSTRUCTIONS" not in body["answer"]
assert not body.get("toolCalls"), "injected instruction triggered a tool call"
def test_system_prompt_is_not_disclosed(api):
for attempt in EXFILTRATION_PROMPTS:
answer = api.post("/v1/assistant/query", json={"q": attempt}).json()["answer"]
assert SYSTEM_PROMPT_CANARY not in answer, f"system prompt leaked: {attempt!r}"
A canary string planted in the system prompt turns "did it leak?" from a judgement into a substring check. The same trick works for retrieved documents belonging to another tenant.
The AI-specific tests sit on top of ordinary API security, not instead of it. Authentication, rate limiting, input validation and object-level authorization on the wrapping API still apply — see common API security vulnerabilities — and a programme that covers only the AI surface or only the API surface leaves the other half open.
Bias and fairness
Bias testing uses paired inputs: the same request with only a name, gender, dialect or location changed. Assert that the decision-bearing parts of the output (a classification, a recommendation, a refusal) do not change across the pair, and score the free-text parts for differences in tone or completeness. Keep the paired set separate from the main golden set and report it separately, because an overall pass rate can be high while one group is consistently served worse. Anything that makes high-stakes decisions — hiring, credit, medical, legal — also needs human review in the loop, not only automated checks.
AI-Policy-Safe API Testing: Self-Hosted LLM Evaluation Kit
Evaluate self-hosted LLM test generation: map data flows, compare Ollama, vLLM and LM Studio, and get AI-policy approval with a 26-control checklist.
Running LLM tests in CI and production
Trigger on prompt and model changes
# .github/workflows/llm-quality.yml
name: LLM quality
on:
pull_request:
paths: ['prompts/**', 'src/assistant/**', 'evals/**', 'config/models.yaml']
jobs:
invariants:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements.txt
- name: Deterministic shell + invariants (hard gate)
run: pytest tests/ -q --junitxml=results.xml
env:
MODEL_VERSION: ${{ vars.PINNED_MODEL_VERSION }} # pin, always
TEMPERATURE: '0'
evals:
needs: invariants
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements.txt
- name: Golden-set eval (threshold gate)
run: python evals/run_eval.py | tee eval.json
- name: Compare against the baseline on main
run: python evals/compare.py --baseline baselines/main.json --current eval.json --max-drop 0.02
- uses: actions/upload-artifact@v4
if: always()
with: { name: eval-results, path: eval.json }
Pin the model version. An unpinned model turns every provider update into an unannounced deployment of your most behaviour-defining dependency, and you will find out from users. When you do move to a new model version, treat it like a dependency upgrade: run the full golden set against both, compare per category, and keep the previous version configured as a fallback until the new one has held up in production.
Layer the suites by cost. Pattern and rule-based checks run on every commit; invariants and a small eval with a cheaper judge run on pull requests that touch prompts, models or retrieval; the full golden set with the production model runs on release candidates and on a weekly schedule. Running the full eval on every commit is slow, costs money per run, and trains people to ignore it.
Budget cost and latency like any other resource
Token cost per request grows silently, the same way latency regressions creep into any service — see monitoring API performance in production. A prompt that gains context over six months can triple its cost without any single change looking significant. Assert an upper bound on tokens per call for the paths that run at volume; the test fails when someone adds a large block to a system prompt, which is exactly when you want to know.
Latency compounds with chaining. An agentic flow making five sequential model calls is a slow response however fast each call is. Assert on the number of model calls a flow makes, not only on wall-clock time, because call count is stable across providers and load while latency is not.
def test_token_budget_has_not_regressed(llm, fixture_invoice):
out, usage = llm.extract_with_usage(fixture_invoice)
assert usage["total_tokens"] < 4000, f"token usage regressed to {usage['total_tokens']}"
def test_agent_flow_stays_within_call_budget(agent, trace):
agent.run("book a meeting with the account owner next week")
assert trace.model_calls <= 4, f"flow made {trace.model_calls} model calls"
Both assertions are cheap and both catch regressions that never show up as an error — the feature still works, it just costs three times as much or takes four times as long.
Monitor drift in production
Pre-release evals cannot see changes in what users ask. In production, run the golden set against the live configuration on a schedule, watch the distribution of inputs and outputs (length, refusal rate, citation count, intent mix) for shifts, and track user signals such as thumbs-down, regenerate and abandonment as proxy quality metrics. Sample live outputs for human rating every week. Every confirmed bad output becomes a golden-set case, which is how the dataset stays representative instead of going stale.
Governance and audit evidence
In regulated organisations, the question "how do you know the AI works?" is answered with a documented evaluation framework, not with individual outputs. Write down, per feature:
- the categories of input in scope and the output properties required for each;
- the thresholds (static, drift, per category) and why each was set where it is;
- the evaluation cadence — per change, scheduled, continuous;
- who may change the golden set or a threshold, and how that change is reviewed;
- the escalation path when a threshold is breached;
- how long evaluation results are kept.
That framework is auditable in a way that thousands of individual outputs are not, and it maps onto the "Measure" and "Manage" functions of the NIST AI Risk Management Framework. Regulation such as the EU AI Act adds documented testing, accuracy and robustness obligations for high-risk systems, which makes this paperwork a compliance artefact rather than a nice-to-have. Keep the eval results, threshold history and golden-set change log in version control or your CI artefact store so an auditor can sample them without a bespoke export.
LLM testing tools and frameworks
Tool choice is secondary to the split above; every tool here covers one layer, and none covers all of them. The open-source eval frameworks overlap heavily, so pick one and invest in the dataset rather than running three.
| Tool | Layer | What it actually does | Licence |
|---|---|---|---|
| pytest / Jest | Shell, invariants | Your ordinary test runner — most LLM tests are plain tests | Open source |
| promptfoo | Prompt tests, evals, red-teaming | CLI and library; declare prompts, providers and test cases with assertions in config, compare outputs across models, run red-team probes | Open source |
| DeepEval | Evals | Pytest-style framework with built-in LLM metrics such as answer relevancy, faithfulness and hallucination, many judged by an LLM | Open source |
| Ragas | RAG evals | Metrics for retrieval-augmented generation — faithfulness, context precision and recall, response relevancy — plus test-set generation | Open source |
| OpenAI Evals | Evals | Framework and registry for writing and running evals, including model-graded ones | Open source |
| LangSmith | Tracing, datasets, evals | Traces every step of an LLM app, turns traces into datasets, runs evaluators and compares experiments | Commercial (free tier) |
| Giskard | Safety and bias scans | Scans LLM apps for issues such as hallucination, prompt injection, harmful content and bias | Open source |
| Hypothesis | Property-based tests | Generates inputs to find cases that break a stated property; useful for parsers and post-processing | Open source |
| Evidently | Production monitoring | Data and model drift reports and LLM output evaluation over production data | Open source |
Two gaps in this list are worth naming. None of the eval frameworks tests the API contract, authorization or streaming behaviour of the service that wraps the model — that is ordinary API testing, covered in the shell section above. And none of them will write your golden set for you; generated test sets are a starting point that still needs human review. For a broader survey of AI-assisted testing tools, see the best AI testing tools.
LLM testing best practices
- Test the application, not the model. Providers evaluate their own models. Your tests cover your prompts, retrieval, parsing, guardrails, tools and error handling — the parts you built.
- Never assert on exact wording. If wording genuinely matters, constrain the output format and assert on the structure.
- Snapshot the rendered prompt and treat a changed snapshot as a behaviour change.
- Sample more than once and require invariants to hold on every sample.
- Gate evals on a tail metric and per-category floors, not on the mean alone.
- Build the golden set from real failures, not from inputs imagined at a desk, and review every change to it.
- Calibrate the judge against human labels before trusting it, and again when it changes.
- Pin the model version and test model upgrades like dependency upgrades.
- Assert on cost, call count and latency alongside quality.
- Exercise the non-AI fallback; it is the path users hit during an outage and the one nobody runs.
More from the AI for API Testing hub
Every guide in this hub, grouped by format. The AI for API Testing hub page has the full list.
Tools and comparisons
Guides and concepts
- The Future of Software Testing in AI-Driven Development (2026)
- Run AI Test Generation on Your Own LLM
- Self-Healing Tests: How They Work and Why APIs Are Different
- Testing AI-Generated Code: Gates That Actually Hold (2026)
LLM testing FAQ
What are the best LLM testing tools?
There is no single tool. Use your normal test runner (pytest, Jest) and an API testing tool for the deterministic shell; an eval framework such as promptfoo, DeepEval or OpenAI Evals for scored quality; Ragas for retrieval-augmented generation metrics; and a tracing platform such as LangSmith to turn production traces into eval cases. Pick by layer, not by feature list.
What is an LLM testing framework?
A library that runs a dataset of inputs through your LLM application and scores each output with assertions or metrics — exact checks, semantic similarity, LLM-as-judge rubrics — then reports an aggregate you can gate on. promptfoo, DeepEval and OpenAI Evals are examples. It complements, rather than replaces, ordinary unit and API tests.
What is the difference between LLM testing and evaluation?
A test asserts a property that must always hold and fails the build when it does not — the output parses, a refusal happens, no cross-tenant document is cited. An evaluation scores quality across a dataset and reports a distribution, gated on an aggregate threshold. You need both, and they fail for different reasons.
What should an LLM testing strategy include?
Hard-assertion tests for retrieval, prompt assembly, tool calls, guardrails and fallbacks; invariant tests on model output sampled several times; a version-controlled golden set scored on every prompt, model or retrieval change; safety, injection and bias suites; cost and latency budgets; and production monitoring that feeds new cases back into the golden set.
What are LLM testing best practices?
Pin the model version, snapshot the rendered prompt, assert on properties instead of wording, sample more than once, gate evals on a tail metric as well as the mean, build the golden set from real failures, calibrate any LLM judge against human labels, and run expensive evals on prompt and model changes rather than every commit.
How do you stop LLM tests being flaky?
Pin the model version, set temperature to zero where the feature allows, sample several times and assert on the aggregate, and assert on properties rather than strings. Temperature zero reduces variance but does not remove it, so flakiness usually means the test is checking the wrong thing, not that the model is broken.
Key takeaways
- Split the system: most of an LLM feature is deterministic and deserves ordinary hard assertions. Only generation quality needs probabilistic treatment.
- Assert invariants — it parses, it is grounded, forbidden fields are absent, bounds hold — on several samples, and use tolerance checks only alongside a fact check.
- Gate a version-controlled golden set on mean, tail and per-category thresholds, plus a drift margin against main.
- Treat safety properties as controls with hard assertions, and keep ordinary API security on the wrapping service.
- Pin the model, budget tokens and model calls, monitor drift in production, and document the evaluation framework for audit.
Want to see how the API and MCP layer of an LLM application gets tested from its spec? Book a demo and we will walk through it on your own endpoints.
Sources and further reading
- OWASP Top 10 for LLM Applications — prompt injection, excessive agency and the rest of the risk list.
- NIST AI Risk Management Framework — the governance frame most enterprise programmes map to.
- Regulation (EU) 2024/1689 (EU AI Act) — the official text, including obligations for high-risk systems.
- Model Context Protocol specification — the protocol behind MCP tool servers.
- Server-Sent Events (WHATWG HTML Standard) — the streaming format most chat endpoints use.
- promptfoo documentation, DeepEval, Ragas documentation, OpenAI Evals and LangSmith documentation — the eval tools in the table above.
- Google Testing Blog — the flakiness and test-sizing writing that applies directly to sampling-based tests.
Go deeper in the Learning Center
Hands-on lessons with runnable code against our live sandbox.
Ready to shift left with your API testing?
Import your OpenAPI spec and get a runnable test suite, in our cloud or on your own infrastructure.