API Testing in Kubernetes: Run Suites In-Cluster (2026)

For the wider strategy this sits in, see testing strategy for cloud-native applications.

The core pattern: a test Job

# k8s/test-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: orders-api-tests
  namespace: staging
spec:
  backoffLimit: 0                 # a flaky retry hides a real failure
  ttlSecondsAfterFinished: 3600   # clean up automatically
  template:
    metadata:
      labels: { app: orders-api-tests }
    spec:
      serviceAccountName: api-tests
      restartPolicy: Never
      containers:
        - name: tests
          image: registry.internal/tools/api-tests:2026.8
          args:
            - schemathesis
            - run
            - /specs/openapi.yaml
            - --url=http://orders-api.staging.svc.cluster.local:8080
            - --checks=all
            - --report=junit
            - --report-junit-path=/results/results.xml
          env:
            - name: API_TOKEN
              valueFrom: { secretKeyRef: { name: api-test-token, key: token } }
          volumeMounts:
            - { name: specs, mountPath: /specs, readOnly: true }
            - { name: results, mountPath: /results }
          resources:
            requests: { cpu: 200m, memory: 256Mi }
            limits:   { cpu: "1",  memory: 512Mi }
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
      volumes:
        - { name: specs, configMap: { name: orders-openapi } }
        - { name: results, emptyDir: {} }

Three details matter more than they look:

• backoffLimit: 0. A Job that retries turns an intermittent failure into a green result. If the suite is flaky, fix the suite.
• The in-cluster DNS name, not the ingress hostname. That is the whole point — you are testing the path a sibling service uses.
• Its own service account. Give the tests exactly the permissions they need, which for most suites is none at all beyond reading their own secret.

RBAC for the test runner

# k8s/test-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata: { name: api-tests, namespace: staging }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: api-tests, namespace: staging }
rules:
  # only what a suite that needs to check pod health actually requires
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: api-tests, namespace: staging }
subjects: [{ kind: ServiceAccount, name: api-tests, namespace: staging }]
roleRef: { kind: Role, name: api-tests, apiGroup: rbac.authorization.k8s.io }

If your suite does not inspect cluster objects, drop the Role entirely. A test runner with cluster-admin is a standing risk for no benefit.

Attaching the suite to a Helm release

A helm.sh/hook: test pod runs on helm test, which makes the smoke suite part of the chart rather than part of a separate pipeline:

# charts/orders-api/templates/tests/smoke.yaml
apiVersion: v1
kind: Pod
metadata:
  name: {{ include "orders-api.fullname" . }}-smoke
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  restartPolicy: Never
  containers:
    - name: smoke
      image: curlimages/curl:8.8.0
      command: ["/bin/sh", "-c"]
      args:
        - |
          set -e
          BASE=http://{{ include "orders-api.fullname" . }}:{{ .Values.service.port }}
          curl -sf "$BASE/health" >/dev/null
          [ "$(curl -so /dev/null -w '%{http_code}' "$BASE/v1/orders")" = 401 ]
          curl -sf "$BASE/openapi.yaml" | head -1 | grep -q openapi

helm upgrade --install orders-api ./charts/orders-api -n staging
helm test orders-api -n staging --logs

Gating a GitOps sync

With Argo CD, a PostSync hook makes the tests part of the deployment rather than something that happens afterwards — if the Job fails, the sync is marked failed:

# k8s/postsync-tests.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: orders-api-postsync-tests
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 0
  template:
    spec:
      serviceAccountName: api-tests
      restartPolicy: Never
      containers:
        - name: tests
          image: registry.internal/tools/api-tests:2026.8
          args: ["schemathesis", "run", "/specs/openapi.yaml",
                 "--url=http://orders-api:8080", "--checks=all"]
          volumeMounts: [{ name: specs, mountPath: /specs, readOnly: true }]
      volumes:
        - { name: specs, configMap: { name: orders-openapi } }

Ephemeral namespaces per pull request

The cleanest isolation model in Kubernetes is a namespace with a lifetime of one pull request. Nothing is shared, so nothing needs resetting between runs:

# .github/workflows/pr-cluster-tests.yml
name: In-cluster API tests
on: pull_request
jobs:
  test:
    runs-on: ubuntu-latest
    env:
      NS: pr-${{ github.event.number }}
    steps:
      - uses: actions/checkout@v4
      - uses: azure/setup-kubectl@v4
      - name: Create the namespace
        run: kubectl create namespace "$NS"

      - name: Deploy the service and its dependencies
        run: |
          helm upgrade --install orders-api ./charts/orders-api \
            -n "$NS" --set image.tag="${{ github.sha }}" --wait --timeout 5m

      - name: Run the suite as a Job and wait
        run: |
          kubectl apply -n "$NS" -f k8s/test-job.yaml
          kubectl wait -n "$NS" --for=condition=complete --timeout=10m job/orders-api-tests \
            || { kubectl logs -n "$NS" job/orders-api-tests; exit 1; }

      - name: Collect results before the namespace goes away
        if: always()
        run: |
          POD=$(kubectl get pod -n "$NS" -l job-name=orders-api-tests -o name | head -1)
          kubectl cp -n "$NS" "${POD#pod/}:/results/results.xml" results.xml || true

      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: api-results, path: results.xml }

      - name: Tear down
        if: always()
        run: kubectl delete namespace "$NS" --wait=false

Copying results out of a pod is the fragile step — the pod has to still exist. If runs matter, push results to object storage from inside the container instead and let the namespace disappear whenever it likes.

Where to run API tests, compared

The first row is where most teams start and the third is where most teams should end up. The rows in between are useful stages rather than destinations.

When to reach for purpose-built tooling

Plain Jobs cover one suite well. What they do not give you is scheduling across many suites, a history of runs, or a shared results view for several teams. That is the gap Testkube and similar tools fill, by modelling tests as Kubernetes custom resources with an operator behind them.

The decision is the usual one: adopt it when the operational cost of the thing you have built by hand exceeds the cost of running someone else's operator — typically somewhere past five teams and twenty suites, not before.

Testing through the service mesh

If the cluster runs Istio or Linkerd, an in-cluster test exercises mesh policy as well as application behaviour — which is the point, and also a source of confusing failures if you do not account for it.

# the test pod needs to be in the mesh, or it is testing a different path
apiVersion: batch/v1
kind: Job
metadata:
  name: orders-api-tests
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
        # a Job with a sidecar never completes unless the sidecar is told to exit
        proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
    spec:
      restartPolicy: Never
      containers: [{ name: tests, image: registry.internal/tools/api-tests:2026.8 }]

Two mesh-specific things worth asserting explicitly, because both fail silently:

def test_mtls_is_enforced_between_services(cluster):
    """A plaintext call from inside the mesh must be rejected."""
    out = cluster.exec_in_pod("debug-pod",
        "curl -s -o /dev/null -w '%{http_code}' http://orders-api:8080/v1/orders")
    assert out.strip() in ("000", "503"), "plaintext accepted — mTLS not enforced"

def test_authorization_policy_blocks_unlisted_callers(cluster):
    out = cluster.exec_in_pod("unrelated-service",
        "curl -s -o /dev/null -w '%{http_code}' http://orders-api:8080/v1/orders")
    assert out.strip() == "403", "AuthorizationPolicy is not restricting callers"

The sidecar-completion detail catches most teams once: a Job whose pod includes a sidecar stays Running forever because the proxy never exits, so kubectl wait --for=condition=complete times out and the pipeline reports a failure that is not one.

Cost, scheduling and cleanup

Ephemeral namespaces are the right isolation model and they will quietly consume a cluster if nothing reaps them.

# a namespace that deletes itself, as a backstop for a pipeline that died mid-run
apiVersion: batch/v1
kind: CronJob
metadata: { name: reap-pr-namespaces, namespace: ops }
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: namespace-reaper
          restartPolicy: Never
          containers:
            - name: reap
              image: bitnami/kubectl:latest
              command: ["/bin/sh", "-c"]
              args:
                - |
                  CUTOFF=$(date -u -d '4 hours ago' +%s)
                  kubectl get ns -l purpose=pr-preview \
                    -o jsonpath='{range .items[*]}{.metadata.name} {.metadata.creationTimestamp}{"\n"}{end}' |
                  while read -r ns created; do
                    [ "$(date -u -d "$created" +%s)" -lt "$CUTOFF" ] && kubectl delete ns "$ns" --wait=false
                  done

Three cost controls that matter more than they sound:

• Set requests and limits on the test container. An unbounded test pod on a shared node is an outage waiting for a busy afternoon.
• Set ttlSecondsAfterFinished on every Job so completed pods do not accumulate.
• Label everything with the pull-request number, so a reaper can find orphans and a human can tell what a stray namespace was for.

A scheduled reaper is not optional. Pipelines get cancelled, runners get killed, and the cleanup step is the one that does not run when something goes wrong — which is exactly when a namespace is left behind.

Sources and further reading

• Kubernetes documentation — Jobs, RBAC, service accounts and namespace lifecycle.
• Testkube — running test suites as Kubernetes-native workloads.
• Testcontainers — the alternative for dependency isolation when tests run outside the cluster.

Key takeaways

• Run the suite in-cluster so it uses the same DNS, network policy and identity as real traffic; an external runner tests a different path.
• Use a Job, not a Pod, and set backoffLimit: 0 — a retrying test job converts flakiness into false confidence.
• Give the runner its own service account with the narrowest role it needs, and usually no cluster access at all.
• Attach the smoke suite to the chart with a Helm test hook, and gate GitOps releases with an Argo CD PostSync hook so failing tests fail the sync.
• Ephemeral namespaces per pull request are the cleanest isolation model; push results out from inside the container rather than copying them from a pod that is about to disappear.

Related articles

Testing Strategy for Cloud-Native Applications | API Testing in CI/CD Pipelines | Microservices Testing: The Complete Guide | Testing Architecture for Scalable Systems | Scalable API Test Reporting