How to Test LLM Applications: A Practical Guide (2026)

For the enterprise-scale framing of the same problem, see testing non-deterministic AI systems.

In this guide

1. Split the system first
2. Test the shell with hard assertions
3. Test the core with invariants
4. Score quality with an eval suite
5. Prompt and model changes are the real regression trigger
6. What to test that teams usually miss
7. Where to spend the testing budget
8. Making non-deterministic tests stable
9. Budget cost and latency like any other resource
10. Common mistakes when testing LLM applications
11. Frequently asked questions about testing LLM applications

Split the system first

Most of the table is deterministic. Teams that skip straight to "how do I evaluate quality?" leave the majority of their failure surface untested. Tool calls in particular are ordinary API surface — contract testing and authentication and authorization testing apply unchanged.

Test the shell with hard assertions

# tests/test_assistant_shell.py

def test_retrieval_never_crosses_the_tenant_boundary(api, tenant_a_token):
    """The RAG layer is deterministic and this is a security control."""
    r = api.post("/v1/assistant/query", json={"q": "revenue last quarter"},
                 headers={"Authorization": f"Bearer {tenant_a_token}"})
    assert r.status_code == 200
    for citation in r.json()["citations"]:
        assert citation["tenantId"] == "tenant-a", "cross-tenant document retrieved"

def test_prompt_template_is_stable(snapshot):
    """A prompt change is a behaviour change — make it reviewable."""
    rendered = build_prompt(question="What is our refund policy?",
                            documents=[FIXTURE_DOC_1, FIXTURE_DOC_2])
    snapshot.assert_match(rendered, "assistant_prompt.txt")

def test_model_timeout_degrades_instead_of_failing(api, slow_model):
    with slow_model(delay_seconds=30):
        r = api.post("/v1/assistant/query", json={"q": "hello"})
    assert r.status_code == 200
    body = r.json()
    assert body["mode"] == "fallback"
    assert body["answer"], "fallback returned nothing"

def test_guardrail_blocks_the_category_it_is_supposed_to(api):
    r = api.post("/v1/assistant/query", json={"q": DISALLOWED_REQUEST})
    assert r.json()["refused"] is True
    assert r.json().get("answer") in (None, ""), "refused but answered anyway"

Snapshot-testing the rendered prompt is the highest-value item in this list and the most often skipped. Prompts are code; a one-word change can move behaviour, and without a snapshot it lands with no diff for a reviewer to read.

Test the core with invariants

Invariants are properties that must hold for every output, regardless of wording:

# tests/test_llm_invariants.py
import json, jsonschema, pytest

SCHEMA = json.load(open("schemas/extraction.json"))

@pytest.mark.parametrize("sample", range(5))     # same input, five samples
def test_extraction_invariants(llm, fixture_invoice, sample):
    out = llm.extract(fixture_invoice)

    # 1. structural — it parses and validates, every time
    jsonschema.validate(out, SCHEMA)

    # 2. grounded — no value that is not present in the source document
    assert out["total"] in fixture_invoice.text, "hallucinated total"
    assert out["vendor"].lower() in fixture_invoice.text.lower(), "hallucinated vendor"

    # 3. bounded — plausible, not merely well-formed
    assert 0 < float(out["total"]) < 1_000_000

    # 4. safe — the fields it was told to redact are absent
    assert not ({"ssn", "card_number"} & set(out))

def test_refusal_is_stable_across_samples(llm):
    """Behaviour that must be deterministic gets tested as deterministic."""
    outs = [llm.answer(PROMPT_INJECTION_DOC) for _ in range(5)]
    assert all(o["refused"] for o in outs), "refusal is not reliable"

Sampling several times and asserting the property holds every time is what converts a flaky test into a meaningful one. If the property only holds four times in five, you have found a real defect, not a testing problem.

Score quality with an eval suite

Evals answer a different question — not "is this valid?" but "is this good, and is it as good as last week?" Building an eval suite that catches regressions covers the dataset and scoring design; the summary below is the minimum.

# evals/run_eval.py
import statistics, json

def score(actual, expected):
    """Combine cheap deterministic signals with a judged one."""
    s = 0.0
    s += 0.4 * (1.0 if expected["answer_key"].lower() in actual["answer"].lower() else 0.0)
    s += 0.3 * (1.0 if set(expected["must_cite"]) <= set(actual["citation_ids"]) else 0.0)
    s += 0.3 * judge_relevance(actual["answer"], expected["question"])   # LLM-as-judge
    return s

results = [score(run(case.question), case.expected) for case in GOLDEN_SET]
mean = statistics.mean(results)
p10 = statistics.quantiles(results, n=10)[0]

print(json.dumps({"n": len(results), "mean": round(mean, 3), "p10": round(p10, 3)}))

# gate on both — a good mean can hide a bad tail
assert mean >= 0.92, f"quality regression: mean {mean:.3f} < 0.92"
assert p10 >= 0.80, f"tail regression: p10 {p10:.3f} < 0.80"

Two rules make eval suites useful rather than decorative:

1. Gate on a tail metric as well as the mean. A mean of 0.93 with a p10 of 0.4 is a feature that fails badly for one user in ten.
2. Build the golden set from real failures. Every production incident becomes a case. A golden set of happy paths measures nothing you were at risk from.

Prompt and model changes are the real regression trigger

# .github/workflows/llm-quality.yml
name: LLM quality
on:
  pull_request:
    paths: ['prompts/**', 'src/assistant/**', 'evals/**', 'config/models.yaml']
jobs:
  invariants:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - name: Deterministic shell + invariants (hard gate)
        run: pytest tests/ -q --junitxml=results.xml
        env:
          MODEL_VERSION: ${{ vars.PINNED_MODEL_VERSION }}   # pin, always
          TEMPERATURE: '0'

  evals:
    needs: invariants
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - name: Golden-set eval (threshold gate)
        run: python evals/run_eval.py | tee eval.json
      - name: Compare against the baseline on main
        run: python evals/compare.py --baseline baselines/main.json --current eval.json --max-drop 0.02
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-results, path: eval.json }

Pin the model version. An unpinned model turns every provider update into an unannounced deployment of your most behaviour-defining dependency, and you will find out from users.

What to test that teams usually miss

• Cost and latency as assertions. A prompt change that doubles token usage is a regression; assert on tokens per request and p99 latency alongside quality.
• The empty and adversarial inputs. Blank question, 50,000-character question, a question in another language, a question containing the retrieval delimiter.
• Citation integrity. If the UI shows sources, assert every cited id exists and that the cited document actually contains the claim.
• Prompt injection through retrieved content, not just through the user's message. That is the realistic attack path for RAG, and it sits alongside the injection and authorization risks in the OWASP API Security Top 10.
• The non-AI fallback. It is the path users will hit during an outage and the one nobody exercises.

def test_token_budget_has_not_regressed(llm, fixture_invoice):
    out, usage = llm.extract_with_usage(fixture_invoice)
    assert usage["total_tokens"] < 4000, f"token usage regressed to {usage['total_tokens']}"

def test_injection_via_retrieved_document_is_ignored(api, poisoned_doc):
    r = api.post("/v1/assistant/query", json={"q": "summarise the report"})
    body = r.json()
    assert "IGNORE PREVIOUS INSTRUCTIONS" not in body["answer"]
    assert not body.get("toolCalls"), "injected instruction triggered a tool call"

Where to spend the testing budget

LLM features attract disproportionate testing effort on the model and disproportionately little on everything else. A rough allocation that matches where defects actually come from:

The pattern behind those numbers: most of what users experience as "the AI got it wrong" is not the model generating badly. It is the model generating reasonably from the wrong context, or generating fine and the surrounding code mishandling it.

A useful diagnostic when a feature is misbehaving in production is to replay the exact retrieved context through the model offline. If the output is good with the right context and bad in production, the defect is in retrieval, and no amount of prompt tuning will fix it — which is the conclusion teams reach after a fortnight of prompt tuning.

Making non-deterministic tests stable

The objection to testing LLM features is that the output changes between runs. That is true, and it is manageable — the trick is to remove the variance you do not need and to assert differently on the variance you cannot remove.

Remove what you can. Pin the model version explicitly; a provider alias like "latest" silently changes what you are testing. Set temperature to zero for anything you assert on deterministically, and supply a seed where the provider supports one. None of this makes output identical run to run, but it removes most of the drift.

Assert on properties, not strings. The stable assertions are structural and semantic, not literal:

# brittle — one rephrasing away from failing
assert response == "Your order #123 shipped on 4 March."

# stable — the properties that actually matter
assert response_json["order_id"] == "123"          # extracted correctly
assert "4 March" in response or "2026-03-04" in response
assert len(response) < 500                          # stayed concise
assert not contains_pii(response)                   # never leaks

Separate the two failure types. A hard assertion that fails is a bug and should block. A quality score that dips is a signal and should be gated on an aggregate threshold across a dataset, not on a single case — one bad generation out of a hundred is noise, and a five-point drop in the mean is not.

Re-run before believing a single failure. For scored tests, a single-run failure at the threshold boundary is usually variance. Run the eval set, gate on the aggregate, and investigate the individual cases that fail consistently.

Budget cost and latency like any other resource

An LLM feature has two operating characteristics that traditional features do not, and both belong in tests rather than in a dashboard discovered later.

Token cost per request grows silently, in the same way latency regressions creep into any service — see monitoring API performance in production. A prompt that gains context over six months triples its cost without any single change looking significant. Assert an upper bound on tokens per call for the paths that run at volume — the test fails when someone adds a large block to a system prompt, which is exactly when you want to know.

Latency compounds with chaining. Each model call adds seconds, and an agentic flow making five sequential calls is a fifteen-second response. Assert on the number of model calls a flow makes, not only on wall-clock time, because call count is stable across providers and load while latency is not.

Both assertions are cheap and both catch the class of regression that never shows up as an error — the feature still works, it just costs three times as much or takes four times as long.

Common mistakes when testing LLM applications

Testing the model instead of the application. Providers evaluate their own models extensively, and the tooling for what remains is surveyed in the best AI testing tools in 2026. Your tests should cover your prompts, your retrieval, your parsing, your guardrails and your error handling — the parts you built.

Asserting on exact wording. Almost always wrong, and the source of most abandoned LLM test suites. If wording genuinely matters, constrain the output format and assert on the structure.

Building the golden dataset from imagined inputs. Datasets written at a desk cover what the author expected. The valuable cases come from real traffic and from failures already observed in production.

No test for the failure path. Providers rate-limit, time out and return malformed output. What the application does then is a hard-assertion test, and it is usually the least-covered part of the system.

Letting the judge go uncalibrated. If you score quality with another model, the judge is itself a component that can be wrong. Check it against human labels on a sample before trusting its verdicts — see the LLM evals guide.

Running evals on every commit. They are slow and they cost money per run. Hard assertions belong on every commit; scored evals belong on a schedule and before a release, or on changes to prompts, models and retrieval.

Frequently asked questions about testing LLM applications

Can you unit test an LLM? You can unit test everything around it and property-test what comes out of it. What you cannot do is assert equality on generated text — the same prompt does not reliably produce the same string, so an equality assertion is a flaky test by construction.

What is the difference between a test and an eval? A test asserts a property that must always hold and fails the build when it does not. An eval scores quality across a dataset and reports a distribution, gated on an aggregate threshold. You need both, and they fail for different reasons.

How large should a golden set be? Large enough that a single bad sample cannot move the aggregate — typically 50 to 200 cases for a focused feature. Weight it toward the failures you have actually seen in production rather than toward happy paths.

Is LLM-as-judge reliable? Reliable enough for relative comparison between versions, unreliable as an absolute score. Use it to detect regression between two builds, calibrate it against human labels periodically, and never let it be the only signal on a safety-critical property.

How do I stop tests being flaky? Pin the model version, set temperature to zero where the feature allows, sample several times and assert on the aggregate, and assert on properties rather than strings. Flakiness usually comes from testing the wrong thing, not from the model.

What should gate a release? Invariant tests should gate hard — schema validity, grounding, absence of forbidden fields, guardrail behaviour. Eval scores should gate on a threshold with both a mean and a tail metric, so an average that hides a bad tail does not pass.

Sources and further reading

• OWASP Top 10 for LLM Applications — prompt injection, excessive agency and the rest of the risk list.
• NIST AI Risk Management Framework — the governance frame most enterprise programmes map to.
• Google Testing Blog — the flakiness and test-sizing writing that applies directly to sampling-based tests.

Key takeaways

• Split the system: most of an LLM feature is deterministic and deserves ordinary hard assertions. Only generation quality needs probabilistic treatment.
• Never assert equality on generated text. Assert invariants — it parses, it is grounded in the source, forbidden fields are absent, bounds hold — and sample several times.
• Snapshot the rendered prompt. Prompts are code, and a prompt change with no diff is a behaviour change no one reviewed.
• Gate evals on a tail metric as well as the mean, and build the golden set from real production failures rather than happy paths.
• Pin the model version. An unpinned model is an unannounced deployment of your most behaviour-defining dependency.
• Assert on cost and latency too — a quality-neutral prompt change that doubles tokens is still a regression.