That change is genuinely useful in some places and actively harmful in others. The distinction is worth getting right before a procurement cycle.

What "agentic" actually adds

The last two rows are the ones that decide whether a tool belongs in a pipeline.

The four questions to ask any vendor

1. Does the agent author or execute?

The safe architecture is: the agent generates cases, a human reviews the diff, and an ordinary runner executes them deterministically. The unsafe one is an agent deciding at gate time what to assert — the same commit can then pass and fail, and nobody can reproduce a failure.

# safe: generation is a separate, reviewable step
# .github/workflows/generate-tests.yml
on:
  pull_request:
    paths: ['openapi.yaml']
jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/agent-generate-tests.sh --spec openapi.yaml --out tests/generated/
      - name: Commit the generated suite for review
        run: |
          git config user.name ci-bot && git config user.email ci-bot@example.com
          git add tests/generated/
          git diff --cached --quiet || git commit -m 'chore: regenerate tests from spec'
          git push

# the gate is a normal, deterministic run
# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/ -q          # no model in this path at all

2. Can I read every generated case?

If the cases live only in the vendor's database and the UI shows you a summary, you cannot review what is being asserted, cannot diff it when it changes, and cannot leave. Ask for the export during the trial.

3. What does it do when the interface changes — and can I see what it changed?

Self-healing is the headline feature and the one with the sharpest edge. A tool that quietly rewrites an assertion so the suite stays green has removed the signal you bought it for. The acceptable behaviour is: propose the repair, show the diff, and let a human accept it.

# what a good repair looks like — reviewable, not silent
Suggested repair  tests/generated/test_orders.py::test_create_order
  Spec change:    Order.status enum gained "on_hold" (openapi.yaml:214)
  Assertion:      - assert body["status"] in {"pending", "paid", "shipped"}
                  + assert body["status"] in {"pending", "paid", "shipped", "on_hold"}
  Confidence:     high (schema-derived)
  [ accept ]  [ reject ]  [ open PR ]

4. How does it avoid encoding current behaviour as correct behaviour?

This is the deepest problem in the category. An agent that observes the API and writes assertions from what it saw will happily assert that a 500 on invalid input is expected, because that is what happened. The only real defence is deriving assertions from a declared contract rather than from observed behaviour:

# derived from what the spec says should happen, not from what the API did
schemathesis run openapi.yaml --url "$STAGING_URL" --checks all
# Falsifying example for POST /v1/orders:
#   {"sku": "", "qty": 0}
#   Response: 500   <- the spec declares 422, so this is a finding, not a baseline

Ask any vendor directly: when the API misbehaves, does your tool report a defect or record a new expectation? The answer separates the category.

Where it works today

It works where a machine-readable contract exists. Generating and maintaining API tests from an OpenAPI document is no longer research — the open-source baseline does it, and commercial tools add coverage tracking, maintenance and history on top. See AI API testing: the complete guide for that end of the category.

It half-works for exploratory discovery: pointing an agent at an application to find states a scripted suite never reaches. Useful for finding candidates, not for gating — treat the output as a bug-hunting session, not a suite.

It does not work yet as a replacement for judgement about what matters. An agent will generate a hundred cases for an endpoint and cannot tell you which three encode a requirement someone would be paged for. That triage is the job that grew, not the one that disappeared.

Keeping determinism when you adopt one

Three rules that hold regardless of vendor:

1. Generated cases are committed artifacts. They live in the repository, they appear in pull requests, and a reviewer signs off. If they cannot be committed, they cannot be a gate.
2. No model call inside the gate. Generation on a schedule or on a spec change; execution with a plain runner. This also keeps the gate fast and free.
3. Measure the suite, not the tool. Mutation score is the honest metric here, because generated suites inflate line coverage without necessarily catching anything:

pip install mutmut
mutmut run --paths-to-mutate app/ --CI
mutmut results --all false | tee mutation.txt
python scripts/assert_mutation_score.py --min 60 mutation.txt

If a generated suite adds 20 points of line coverage and zero points of mutation score, it is testing that the code runs, not that it is correct.

What to measure once one is in place

Adopting an agentic tool changes which metrics are meaningful, and teams that keep reporting the old ones conclude it worked when it did not.

# the two numbers to watch together after adoption
mutmut run --paths-to-mutate app/ --CI
mutmut results --all false | tee mutation.txt
python scripts/assert_mutation_score.py --min 60 mutation.txt

and the one that decides whether anyone keeps using it

pytest --collect-only -q | tail -1 # case count pytest -q --durations=10 # where the runtime went

If case count triples and mutation score is flat, the tool produced work rather than confidence. That is the honest six-week check, and it is worth agreeing on before the trial starts rather than after.

## A staged adoption that does not break the pipeline

The teams that get value from this category all follow roughly the same sequence, and the ones that do not usually skipped straight to stage four.

**Stage 1 — shadow.** Run the tool against your spec or application, commit nothing, and read the output. You are answering one question: are these findings real? Triage 50 of them by hand and compute precision.

**Stage 2 — advisory.** Generate cases into the repository and run them in a non-blocking job. Failures create a report, not a red build. This surfaces flakiness and false positives at zero risk.

```yaml
# .github/workflows/generated-advisory.yml
jobs:
  generated:
    runs-on: ubuntu-latest
    continue-on-error: true          # advisory only, for at least two sprints
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/generated -q --junitxml=generated.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: generated-results, path: generated.xml }

Stage 3 — promote the stable subset. Move the generated cases that have been green and meaningful for two sprints into the blocking suite. Leave the rest advisory.

Stage 4 — regenerate on contract change. Wire generation to spec changes so the suite and the contract move together, with the diff reviewed in the same pull request.

The discipline that makes this work is that stage 2 is allowed to fail. A tool that cannot survive two sprints of advisory running without producing noise nobody trusts has told you something useful for the price of a trial.

Sources and further reading

• OWASP Top 10 for LLM Applications — excessive agency and over-reliance, both directly relevant when an agent touches your pipeline.
• Schemathesis documentation — the open-source, contract-derived baseline any commercial generator has to beat.
• Google Testing Blog — the writing on test sizing and flakiness that agentic suites re-encounter at scale.

Key takeaways

• Agentic QA means the agent decides what to test; the human moves to reviewing. That is the whole change, and it is real.
• The safe architecture is: agent authors, human reviews the diff, deterministic runner executes. A model inside the gate makes builds non-reproducible.
• Demand an export of every generated case. Cases that exist only in a vendor's database cannot be reviewed, diffed or migrated.
• Self-healing must propose and show a diff, never silently rewrite an assertion — a tool that keeps the suite green by editing expectations has removed the signal.
• The category's deepest failure is encoding observed behaviour as correct behaviour. Contract-derived generation avoids it; observation-derived generation does not.
• Judge the result with mutation score, not line coverage. Generated suites are very good at raising the number that means least.

Related articles

AI API Testing: The Complete Guide | How Self-Healing API Tests Work | LLM Evals: A Practical Guide | Testing AI-Generated Code | Best No-Code Test Automation Tools