API Testing in Kubernetes: Run Suites In-Cluster (2026)

For the wider strategy this sits in, see testing strategy for cloud-native applications.

In this guide

1. The core pattern: a test Job
2. RBAC for the test runner
3. Attaching the suite to a Helm release
4. Gating a GitOps sync
5. Ephemeral namespaces per pull request
6. Where to run API tests, compared
7. When to reach for purpose-built tooling
8. Testing through the service mesh
9. Cost, scheduling and cleanup
10. Debugging a test Job that failed
11. Cleaning up after yourself
12. Frequently asked questions about API testing in Kubernetes

The core pattern: a test Job

# k8s/test-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: orders-api-tests
  namespace: staging
spec:
  backoffLimit: 0                 # a flaky retry hides a real failure
  ttlSecondsAfterFinished: 3600   # clean up automatically
  template:
    metadata:
      labels: { app: orders-api-tests }
    spec:
      serviceAccountName: api-tests
      restartPolicy: Never
      containers:
        - name: tests
          image: registry.internal/tools/api-tests:2026.8
          args:
            - schemathesis
            - run
            - /specs/openapi.yaml
            - --url=http://orders-api.staging.svc.cluster.local:8080
            - --checks=all
            - --report=junit
            - --report-junit-path=/results/results.xml
          env:
            - name: API_TOKEN
              valueFrom: { secretKeyRef: { name: api-test-token, key: token } }
          volumeMounts:
            - { name: specs, mountPath: /specs, readOnly: true }
            - { name: results, mountPath: /results }
          resources:
            requests: { cpu: 200m, memory: 256Mi }
            limits:   { cpu: "1",  memory: 512Mi }
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
      volumes:
        - { name: specs, configMap: { name: orders-openapi } }
        - { name: results, emptyDir: {} }

Three details matter more than they look:

• backoffLimit: 0. A Job that retries turns an intermittent failure into a green result. If the suite is flaky, fix the suite.
• The in-cluster DNS name, not the ingress hostname. That is the whole point — you are testing the path a sibling service uses.
• Its own service account. Give the tests exactly the permissions they need, which for most suites is none at all beyond reading their own secret. Secret handling for test runners is covered in SSO and secret management for enterprise API testing.

RBAC for the test runner

# k8s/test-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata: { name: api-tests, namespace: staging }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: api-tests, namespace: staging }
rules:
  # only what a suite that needs to check pod health actually requires
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: api-tests, namespace: staging }
subjects: [{ kind: ServiceAccount, name: api-tests, namespace: staging }]
roleRef: { kind: Role, name: api-tests, apiGroup: rbac.authorization.k8s.io }

If your suite does not inspect cluster objects, drop the Role entirely. A test runner with cluster-admin is a standing risk for no benefit.

Attaching the suite to a Helm release

A helm.sh/hook: test pod runs on helm test, which makes the smoke suite part of the chart rather than part of a separate pipeline:

# charts/orders-api/templates/tests/smoke.yaml
apiVersion: v1
kind: Pod
metadata:
  name: {{ include "orders-api.fullname" . }}-smoke
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  restartPolicy: Never
  containers:
    - name: smoke
      image: curlimages/curl:8.8.0
      command: ["/bin/sh", "-c"]
      args:
        - |
          set -e
          BASE=http://{{ include "orders-api.fullname" . }}:{{ .Values.service.port }}
          curl -sf "$BASE/health" >/dev/null
          [ "$(curl -so /dev/null -w '%{http_code}' "$BASE/v1/orders")" = 401 ]
          curl -sf "$BASE/openapi.yaml" | head -1 | grep -q openapi

helm upgrade --install orders-api ./charts/orders-api -n staging
helm test orders-api -n staging --logs

Gating a GitOps sync

With Argo CD, a PostSync hook makes the tests part of the deployment rather than something that happens afterwards — if the Job fails, the sync is marked failed:

# k8s/postsync-tests.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: orders-api-postsync-tests
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 0
  template:
    spec:
      serviceAccountName: api-tests
      restartPolicy: Never
      containers:
        - name: tests
          image: registry.internal/tools/api-tests:2026.8
          args: ["schemathesis", "run", "/specs/openapi.yaml",
                 "--url=http://orders-api:8080", "--checks=all"]
          volumeMounts: [{ name: specs, mountPath: /specs, readOnly: true }]
      volumes:
        - { name: specs, configMap: { name: orders-openapi } }

Ephemeral namespaces per pull request

The cleanest isolation model in Kubernetes is a namespace with a lifetime of one pull request. Nothing is shared, so nothing needs resetting between runs — which sidesteps most of the problems in managing test data in microservices:

# .github/workflows/pr-cluster-tests.yml
name: In-cluster API tests
on: pull_request
jobs:
  test:
    runs-on: ubuntu-latest
    env:
      NS: pr-${{ github.event.number }}
    steps:
      - uses: actions/checkout@v4
      - uses: azure/setup-kubectl@v4
      - name: Create the namespace
        run: kubectl create namespace "$NS"

      - name: Deploy the service and its dependencies
        run: |
          helm upgrade --install orders-api ./charts/orders-api \
            -n "$NS" --set image.tag="${{ github.sha }}" --wait --timeout 5m

      - name: Run the suite as a Job and wait
        run: |
          kubectl apply -n "$NS" -f k8s/test-job.yaml
          kubectl wait -n "$NS" --for=condition=complete --timeout=10m job/orders-api-tests \
            || { kubectl logs -n "$NS" job/orders-api-tests; exit 1; }

      - name: Collect results before the namespace goes away
        if: always()
        run: |
          POD=$(kubectl get pod -n "$NS" -l job-name=orders-api-tests -o name | head -1)
          kubectl cp -n "$NS" "${POD#pod/}:/results/results.xml" results.xml || true

      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: api-results, path: results.xml }

      - name: Tear down
        if: always()
        run: kubectl delete namespace "$NS" --wait=false

Copying results out of a pod is the fragile step — the pod has to still exist. If runs matter, push results to object storage from inside the container instead and let the namespace disappear whenever it likes.

Where to run API tests, compared

The first row is where most teams start and the third is where most teams should end up. The rows in between are useful stages rather than destinations. For how this fits a wider service-level plan, see a microservices testing strategy.

When to reach for purpose-built tooling

Plain Jobs cover one suite well. What they do not give you is scheduling across many suites, a history of runs, or a shared results view for several teams — the platform problem described in a testing strategy for platform engineering teams. That is the gap Testkube and similar tools fill, by modelling tests as Kubernetes custom resources with an operator behind them.

The decision is the usual one: adopt it when the operational cost of the thing you have built by hand exceeds the cost of running someone else's operator — typically somewhere past five teams and twenty suites, not before.

Testing through the service mesh

If the cluster runs Istio or Linkerd, an in-cluster test exercises mesh policy as well as application behaviour — which is the point, and also a source of confusing failures if you do not account for it. Distributed tracing for microservices is what makes those failures readable.

# the test pod needs to be in the mesh, or it is testing a different path
apiVersion: batch/v1
kind: Job
metadata:
  name: orders-api-tests
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
        # a Job with a sidecar never completes unless the sidecar is told to exit
        proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
    spec:
      restartPolicy: Never
      containers: [{ name: tests, image: registry.internal/tools/api-tests:2026.8 }]

Two mesh-specific things worth asserting explicitly, because both fail silently:

def test_mtls_is_enforced_between_services(cluster):
    """A plaintext call from inside the mesh must be rejected."""
    out = cluster.exec_in_pod("debug-pod",
        "curl -s -o /dev/null -w '%{http_code}' http://orders-api:8080/v1/orders")
    assert out.strip() in ("000", "503"), "plaintext accepted — mTLS not enforced"

def test_authorization_policy_blocks_unlisted_callers(cluster):
    out = cluster.exec_in_pod("unrelated-service",
        "curl -s -o /dev/null -w '%{http_code}' http://orders-api:8080/v1/orders")
    assert out.strip() == "403", "AuthorizationPolicy is not restricting callers"

The sidecar-completion detail catches most teams once: a Job whose pod includes a sidecar stays Running forever because the proxy never exits, so kubectl wait --for=condition=complete times out and the pipeline reports a failure that is not one.

Cost, scheduling and cleanup

Ephemeral namespaces are the right isolation model and they will quietly consume a cluster if nothing reaps them.

# a namespace that deletes itself, as a backstop for a pipeline that died mid-run
apiVersion: batch/v1
kind: CronJob
metadata: { name: reap-pr-namespaces, namespace: ops }
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: namespace-reaper
          restartPolicy: Never
          containers:
            - name: reap
              image: bitnami/kubectl:latest
              command: ["/bin/sh", "-c"]
              args:
                - |
                  CUTOFF=$(date -u -d '4 hours ago' +%s)
                  kubectl get ns -l purpose=pr-preview \
                    -o jsonpath='{range .items[*]}{.metadata.name} {.metadata.creationTimestamp}{"\n"}{end}' |
                  while read -r ns created; do
                    [ "$(date -u -d "$created" +%s)" -lt "$CUTOFF" ] && kubectl delete ns "$ns" --wait=false
                  done

Three cost controls that matter more than they sound:

• Set requests and limits on the test container. An unbounded test pod on a shared node is an outage waiting for a busy afternoon.
• Set ttlSecondsAfterFinished on every Job so completed pods do not accumulate.
• Label everything with the pull-request number, so a reaper can find orphans and a human can tell what a stray namespace was for.

A scheduled reaper is not optional. Pipelines get cancelled, runners get killed, and the cleanup step is the one that does not run when something goes wrong — which is exactly when a namespace is left behind.

Debugging a test Job that failed

A test suite that fails on a laptop gives you a stack trace. A test Job that fails in a cluster frequently gives you nothing useful, because the container never got as far as running the suite. The general method is in how to debug failed API tests in CI/CD; the layers below are the Kubernetes-specific part. Work through the layers in order — most failures are resolved in the first two.

# 1. did the Job create a Pod, and what state is it in?
kubectl get pods -l job-name=api-tests -o wide

# 2. events explain scheduling, image and mount failures — logs never will
kubectl describe pod -l job-name=api-tests | sed -n '/Events:/,$p'

# 3. only now are logs meaningful
kubectl logs -l job-name=api-tests --tail=200

# 4. if the container restarted, the useful output is in the previous one
kubectl logs -l job-name=api-tests --previous

The state in step 1 usually names the problem. ImagePullBackOff is registry credentials or a bad tag. CreateContainerConfigError is almost always a missing Secret or ConfigMap key. Pending is scheduling — insufficient resources, or a node selector nothing matches. OOMKilled in step 2 means the suite needs a higher memory limit; test runners holding large fixtures hit this more often than the application does.

The failure worth calling out separately is the suite running before its dependency is ready. A Job that starts the instant it is created will race a service that takes twenty seconds to accept connections, and the resulting failure looks like a broken test rather than a broken wait:

spec:
  template:
    spec:
      restartPolicy: Never
      initContainers:
        - name: wait-for-api
          image: curlimages/curl:8.6.0
          command: ['sh', '-c', 'until curl -sf http://orders-api/health; do sleep 2; done']
      containers:
        - name: tests
          image: registry.example.com/api-tests:1.4.2

An init container that blocks on a health check turns a race into a deterministic wait, and it fails with a clear timeout rather than an assertion error.

Cleaning up after yourself

Test Jobs accumulate. Each completed Job keeps its Pod for log retrieval, and a suite running per pull request produces them faster than anyone deletes them — eventually consuming namespace quota and making kubectl get pods unusable.

ttlSecondsAfterFinished handles it without a cron job or a cleanup script:

spec:
  ttlSecondsAfterFinished: 3600   # Job and its Pods removed an hour after finishing
  backoffLimit: 0                 # a failing suite is a result, not something to retry

Set backoffLimit: 0 deliberately. The default retries a failed Job, which for a test suite means a flaky test gets a second chance and reports success — exactly the behaviour that erodes trust in a gate. If retries are genuinely wanted, put them in the test runner where they are visible in the report, not in the Job spec where they are silent.

Frequently asked questions about API testing in Kubernetes

Why run API tests inside the cluster instead of from CI? Because an external runner tests a different path. In-cluster tests resolve service DNS, traverse the same network policies and service mesh, and authenticate with the same service account model as real traffic — so misconfigurations in any of those surface before release rather than after.

Should I use a Job or a Pod? A Job. It has retry semantics, a completion status the pipeline can wait on, and ttlSecondsAfterFinished for cleanup. A bare Pod gives you none of that.

How do I get results out of an ephemeral Job? Either stream them to an object store or a reporting service from inside the container, or have the pipeline follow the logs and copy the artifact out before the TTL expires. The first is more reliable.

What is a Helm test hook? A pod annotated with helm.sh/hook: test that runs when you invoke helm test against a release. It is the simplest way to attach a smoke suite to a chart without extra tooling.

Do I need Testkube or a similar tool? Not to start. Plain Jobs cover a single suite well. Purpose-built tooling earns its place when you have many suites across many teams and want scheduling, history and a shared results view without building them.

How do I isolate test runs from each other? Give every pull request its own namespace, deploy the service and its dependencies into it, run the suite, then delete the namespace. Nothing is shared, so nothing needs resetting.

Sources and further reading

• Kubernetes documentation — Jobs, RBAC, service accounts and namespace lifecycle.
• Testkube — running test suites as Kubernetes-native workloads.
• Testcontainers — the alternative for dependency isolation when tests run outside the cluster.

Key takeaways

• Run the suite in-cluster so it uses the same DNS, network policy and identity as real traffic; an external runner tests a different path.
• Use a Job, not a Pod, and set backoffLimit: 0 — a retrying test job converts flakiness into false confidence.
• Give the runner its own service account with the narrowest role it needs, and usually no cluster access at all.
• Attach the smoke suite to the chart with a Helm test hook, and gate GitOps releases with an Argo CD PostSync hook so failing tests fail the sync.
• Ephemeral namespaces per pull request are the cleanest isolation model; push results out from inside the container rather than copying them from a pod that is about to disappear.