API Monitoring and Observability: Synthetic Checks, Signals and Alerts

This is lesson 7.7, the last lesson of API Testing in CI/CD, Regression and Performance. Lessons 7.1 to 7.6 tested the API before it shipped. This one is about after: how you know it is still working, and how you find out why when it isn't.

What is API monitoring and observability?

API monitoring and observability are two answers to one question: is the API working for the people who use it right now? Monitoring checks known conditions continuously (is it up, are errors under 1 percent, is p95 under 300 ms?) and alerts when one is broken. Observability is the ability to ask new questions of a running system from the data it emits (metrics, logs and traces), so you can work out why it broke, including in ways nobody predicted. Monitoring tells you something is wrong; observability helps you find what.

Monitoring and observability side by side

You need both. Monitoring without observability pages you at 3 a.m. with nothing to go on. Observability without monitoring means nobody looks until a customer calls.

Two ways to watch an API: synthetic and real traffic

Synthetic monitoring calls the API on a schedule, from outside, as a client would, and checks the answer. It works when traffic is low (at night, on a new endpoint) and it tests from the client's side of the network. Its limit: it only checks what you scripted.

Real-traffic monitoring measures the requests the API actually serves, from the service itself, its gateway or its load balancer. It sees every client and every endpoint, but only when there is traffic, and only from the server's side.

A synthetic check is a short API test. Here is one, healthy, then on a slow day:

curl -X GET 'https://demo.totalshiftleft.ai/api/v1/products'

curl -X GET 'https://demo.totalshiftleft.ai/api/v1/products?delay=2000'

Both return 200. A check that only asserts the status passes both times, while users wait two seconds. A useful synthetic check asserts three things: the status, one or two fields that prove the answer is real (a non-empty list, a known product's name), and a latency budget.

Which signals to watch

For the traffic an API serves, two short lists cover most needs:

• RED, per endpoint: Rate (requests a second), Errors (failed requests a second, or the share that failed) and Duration (latency percentiles, not averages).
• The four golden signals, per service: latency, traffic, errors and saturation, where saturation is how full the service is (CPU, memory, connection pools, queue depth).

Watch the error rate by status class. A rise in 4xx often means a client or a contract changed; a rise in 5xx means the API or something behind it broke. And split latency by endpoint: one slow search endpoint can hide inside a healthy overall p95.

Alerts worth keeping

Most monitoring setups fail by alerting too much, not too little. A few rules keep alerts meaningful:

1. Alert on symptoms users feel, such as errors and latency, not on causes such as high CPU, which belong on a dashboard.
2. Require persistence. Two or three consecutive failed checks, or a burn rate over several minutes, not one bad sample.
3. Tie alerts to objectives. An SLO such as "99.9 percent of requests succeed over 30 days" gives an error budget; alert when the budget is burning fast, not on every blip.
4. Every alert needs an owner and a first step, written down. An alert nobody knows how to act on gets muted.

API monitoring tools: what each kind does

API monitoring tools fall into three groups, and most teams combine them:

Among API monitoring tools open-source options cover the full stack, at the price of running and maintaining them yourself. OpenTelemetry deserves a special mention: it is the vendor-neutral standard for emitting telemetry, so instrumenting with it keeps you free to change the backend later.

In Shift-Left API: scheduled smoke packs, honestly scoped

Be clear about what Shift-Left API is and isn't here. It is an API testing platform, not an APM or observability tool: it doesn't collect your production traffic, traces, logs or server metrics, and it won't replace Prometheus, Grafana or OpenTelemetry. What it does well is the synthetic half: it runs the API tests you already have on a schedule, as checks, and tells you when they fail. Packs and schedules need the Professional or Enterprise edition, or the 15-day trial.

1. Make a smoke pack. In Execution, build a Test Run Pack of three read-only tests on the sandbox: the product list, the Wireless Mouse, an order with ?expand=user,product. Give the product list test a response-time check, so a slow answer fails it.
2. Notify on failure. On step four, turn on a notification for failed runs, to email, Slack, Microsoft Teams or a webhook. Each message links to that run's report.
3. Schedule it. On step five, set Repeat to Every X Minutes with a value of 15, and schedule it.

The scheduler checks for due packs every 30 seconds and runs them on the Shift-Left API server, so the server has to be running. Each run lands in Reporting with the execution type Automated. When a run fails, its report groups the failures into causes with a verdict, so "the target could not be reached" reads differently from "a bug in the API".

Over time, the runs feed the views from lesson 9.12. On the trial and Enterprise, Analytics → Dashboards shows pass rate, failing tests and each environment per role, with TV Mode for a wall screen, and alert rules on the pass rate, failure count or p95 response time, checked every five minutes and sent in-app, by email or by webhook.

Two cautions. Point a production smoke pack only at read-only tests, with a dedicated account: a scheduled test that creates orders creates real orders. And keep the interval sensible; every 15 minutes is plenty for the shared public sandbox. The scheduler page lists every repeat option, from once to a custom cron expression.

Try it yourself

On a trial, put three sandbox read tests in a Test Run Pack, turn on a notification for failed runs, set Repeat to Every X Minutes with 15, and schedule it. Leave your Shift-Left API server running for an hour, then open Reporting: the pack's runs appear about every 15 minutes, marked Automated. Pause the schedule with the switch on the pack's row when you're done.

Recap

• API monitoring and observability: monitoring alerts on known conditions; observability lets you ask why from metrics, logs and traces.
• What is API monitoring in practice: synthetic checks from outside plus real-traffic signals (RED and the golden signals), with alerts on symptoms that persist.
• API monitoring tools come in three kinds; open-source options cover all of them, and a scheduled smoke pack in Shift-Left API covers the synthetic part with the tests you already have.

For production metrics, SLOs and alerting in depth, read monitoring API performance in production. That completes this track. Next, start the AI and shift-left track with shift left testing, by example.