How to Test LLM Applications: A Practical Guide (2026)

For the enterprise-scale framing of the same problem, see testing non-deterministic AI systems.

Split the system first

Most of the table is deterministic. Teams that skip straight to "how do I evaluate quality?" leave the majority of their failure surface untested.

Test the shell with hard assertions

# tests/test_assistant_shell.py

def test_retrieval_never_crosses_the_tenant_boundary(api, tenant_a_token):
    """The RAG layer is deterministic and this is a security control."""
    r = api.post("/v1/assistant/query", json={"q": "revenue last quarter"},
                 headers={"Authorization": f"Bearer {tenant_a_token}"})
    assert r.status_code == 200
    for citation in r.json()["citations"]:
        assert citation["tenantId"] == "tenant-a", "cross-tenant document retrieved"

def test_prompt_template_is_stable(snapshot):
    """A prompt change is a behaviour change — make it reviewable."""
    rendered = build_prompt(question="What is our refund policy?",
                            documents=[FIXTURE_DOC_1, FIXTURE_DOC_2])
    snapshot.assert_match(rendered, "assistant_prompt.txt")

def test_model_timeout_degrades_instead_of_failing(api, slow_model):
    with slow_model(delay_seconds=30):
        r = api.post("/v1/assistant/query", json={"q": "hello"})
    assert r.status_code == 200
    body = r.json()
    assert body["mode"] == "fallback"
    assert body["answer"], "fallback returned nothing"

def test_guardrail_blocks_the_category_it_is_supposed_to(api):
    r = api.post("/v1/assistant/query", json={"q": DISALLOWED_REQUEST})
    assert r.json()["refused"] is True
    assert r.json().get("answer") in (None, ""), "refused but answered anyway"

Snapshot-testing the rendered prompt is the highest-value item in this list and the most often skipped. Prompts are code; a one-word change can move behaviour, and without a snapshot it lands with no diff for a reviewer to read.

Test the core with invariants

Invariants are properties that must hold for every output, regardless of wording:

# tests/test_llm_invariants.py
import json, jsonschema, pytest

SCHEMA = json.load(open("schemas/extraction.json"))

@pytest.mark.parametrize("sample", range(5))     # same input, five samples
def test_extraction_invariants(llm, fixture_invoice, sample):
    out = llm.extract(fixture_invoice)

    # 1. structural — it parses and validates, every time
    jsonschema.validate(out, SCHEMA)

    # 2. grounded — no value that is not present in the source document
    assert out["total"] in fixture_invoice.text, "hallucinated total"
    assert out["vendor"].lower() in fixture_invoice.text.lower(), "hallucinated vendor"

# 3. bounded — plausible, not merely well-formed
assert 0 < float(out["total"]) < 1_000_000

# 4. safe — the fields it was told to redact are absent
assert not ({"ssn", "card_number"} & set(out))

def test_refusal_is_stable_across_samples(llm): """Behaviour that must be deterministic gets tested as deterministic.""" outs = [llm.answer(PROMPT_INJECTION_DOC) for _ in range(5)] assert all(o["refused"] for o in outs), "refusal is not reliable"

Sampling several times and asserting the property holds every time is what converts a flaky test into a meaningful one. If the property only holds four times in five, you have found a real defect, not a testing problem.

## Score quality with an eval suite

Evals answer a different question — not "is this valid?" but "is this good, and is it as good as last week?"

```python
# evals/run_eval.py
import statistics, json

def score(actual, expected):
    """Combine cheap deterministic signals with a judged one."""
    s = 0.0
    s += 0.4 * (1.0 if expected["answer_key"].lower() in actual["answer"].lower() else 0.0)
    s += 0.3 * (1.0 if set(expected["must_cite"]) <= set(actual["citation_ids"]) else 0.0)
    s += 0.3 * judge_relevance(actual["answer"], expected["question"])   # LLM-as-judge
    return s

results = [score(run(case.question), case.expected) for case in GOLDEN_SET]
mean = statistics.mean(results)
p10 = statistics.quantiles(results, n=10)[0]

print(json.dumps({"n": len(results), "mean": round(mean, 3), "p10": round(p10, 3)}))

# gate on both — a good mean can hide a bad tail
assert mean >= 0.92, f"quality regression: mean {mean:.3f} < 0.92"
assert p10 >= 0.80, f"tail regression: p10 {p10:.3f} < 0.80"

Two rules make eval suites useful rather than decorative:

1. Gate on a tail metric as well as the mean. A mean of 0.93 with a p10 of 0.4 is a feature that fails badly for one user in ten.
2. Build the golden set from real failures. Every production incident becomes a case. A golden set of happy paths measures nothing you were at risk from.

Prompt and model changes are the real regression trigger

# .github/workflows/llm-quality.yml
name: LLM quality
on:
  pull_request:
    paths: ['prompts/**', 'src/assistant/**', 'evals/**', 'config/models.yaml']
jobs:
  invariants:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - name: Deterministic shell + invariants (hard gate)
        run: pytest tests/ -q --junitxml=results.xml
        env:
          MODEL_VERSION: ${{ vars.PINNED_MODEL_VERSION }}   # pin, always
          TEMPERATURE: '0'

  evals:
    needs: invariants
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - name: Golden-set eval (threshold gate)
        run: python evals/run_eval.py | tee eval.json
      - name: Compare against the baseline on main
        run: python evals/compare.py --baseline baselines/main.json --current eval.json --max-drop 0.02
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-results, path: eval.json }

Pin the model version. An unpinned model turns every provider update into an unannounced deployment of your most behaviour-defining dependency, and you will find out from users.

What to test that teams usually miss

• Cost and latency as assertions. A prompt change that doubles token usage is a regression; assert on tokens per request and p99 latency alongside quality.
• The empty and adversarial inputs. Blank question, 50,000-character question, a question in another language, a question containing the retrieval delimiter.
• Citation integrity. If the UI shows sources, assert every cited id exists and that the cited document actually contains the claim.
• Prompt injection through retrieved content, not just through the user's message. That is the realistic attack path for RAG.
• The non-AI fallback. It is the path users will hit during an outage and the one nobody exercises.

def test_token_budget_has_not_regressed(llm, fixture_invoice):
    out, usage = llm.extract_with_usage(fixture_invoice)
    assert usage["total_tokens"] < 4000, f"token usage regressed to {usage['total_tokens']}"

def test_injection_via_retrieved_document_is_ignored(api, poisoned_doc):
    r = api.post("/v1/assistant/query", json={"q": "summarise the report"})
    body = r.json()
    assert "IGNORE PREVIOUS INSTRUCTIONS" not in body["answer"]
    assert not body.get("toolCalls"), "injected instruction triggered a tool call"

Where to spend the testing budget

LLM features attract disproportionate testing effort on the model and disproportionately little on everything else. A rough allocation that matches where defects actually come from:

The pattern behind those numbers: most of what users experience as "the AI got it wrong" is not the model generating badly. It is the model generating reasonably from the wrong context, or generating fine and the surrounding code mishandling it.

A useful diagnostic when a feature is misbehaving in production is to replay the exact retrieved context through the model offline. If the output is good with the right context and bad in production, the defect is in retrieval, and no amount of prompt tuning will fix it — which is the conclusion teams reach after a fortnight of prompt tuning.

Sources and further reading

• OWASP Top 10 for LLM Applications — prompt injection, excessive agency and the rest of the risk list.
• NIST AI Risk Management Framework — the governance frame most enterprise programmes map to.
• Google Testing Blog — the flakiness and test-sizing writing that applies directly to sampling-based tests.

Key takeaways

• Split the system: most of an LLM feature is deterministic and deserves ordinary hard assertions. Only generation quality needs probabilistic treatment.
• Never assert equality on generated text. Assert invariants — it parses, it is grounded in the source, forbidden fields are absent, bounds hold — and sample several times.
• Snapshot the rendered prompt. Prompts are code, and a prompt change with no diff is a behaviour change no one reviewed.
• Gate evals on a tail metric as well as the mean, and build the golden set from real production failures rather than happy paths.
• Pin the model version. An unpinned model is an unannounced deployment of your most behaviour-defining dependency.
• Assert on cost and latency too — a quality-neutral prompt change that doubles tokens is still a regression.

Related articles

Testing Non-Deterministic AI Systems | LLM Evals: A Practical Guide | Testing Strategy for AI-Powered Applications | How to Test MCP Servers | Testing AI-Generated Code