Monitoring Patterns for LLM Systems

Traditional application monitoring catches errors and latency. LLM monitoring must also catch quality - because the most dangerous LLM failures produce no errors at all.

Why LLM Monitoring Is Different

When a traditional API fails, it returns an error code. When an LLM fails, it often returns a 200 OK with a plausible-sounding but wrong answer. This is the fundamental challenge of LLM observability: the failure mode you most need to catch is the one that looks like success to your infrastructure.

A customer service bot that hallucinates a non-existent return policy will report a perfect HTTP success rate. A document summarizer that silently drops a key clause will have excellent latency metrics. A code generation tool that produces subtly broken code will show green dashboards everywhere except where it matters: in the hands of users.

Effective LLM monitoring requires a layer above the infrastructure metrics - a quality layer that treats the content of responses as a first-class signal, not a bonus feature.

The 5 Signals You Must Always Monitor

Signal 1: Latency (P50, P95, P99). Latency for LLM systems is fundamentally different from latency for deterministic APIs. P50 latency may be perfectly acceptable while P99 latency is completely unusable - because long inputs, complex reasoning tasks, and high-concurrency moments all push latency into a long tail. You must track all three percentiles. A P99 that is 5× the P50 is a sign of structural problems (input length variance, context window pressure) that will cause user complaints even if the average experience looks fine.

Signal 2: Error Rate. This includes API errors from the vendor (rate limits, timeouts, model overloads), output format validation failures (the response came back but wasn’t parseable), and fallback triggers (circuit breaker opened, retry limit exceeded). Track each category separately - a spike in format validation failures means something different than a spike in rate limit errors.

Signal 3: Token Usage Per Request. Token usage is both a cost metric and a quality signal. A sudden increase in average token usage often means something changed in the input pipeline (longer user queries, expanded context injection, a bug that duplicates content). A decrease in output token usage may mean the model is being cut off at the max_tokens limit more often. Track input tokens, output tokens, and total tokens separately, and alert on week-over-week deviations >25%.

Signal 4: Quality Score / Hallucination Rate. This is the signal most teams skip - and the most important one to build. At a minimum, implement format compliance rate: what fraction of responses match the expected output format? For higher-stakes systems, implement a judge LLM that scores responses against a rubric, or use deterministic checks (does the response include required fields, does it cite a source within the context, does it avoid forbidden phrases). For systems where ground-truth comparison is possible, track accuracy against a held-out evaluation set on a weekly basis.

Signal 5: User Satisfaction Proxy. Even without explicit thumbs-up/thumbs-down feedback, behavioral signals reveal satisfaction: regeneration rate (did the user immediately ask the same question again?), session abandonment after an LLM response, follow-up question rate (did the user need to ask multiple clarifying questions where one answer should have sufficed?), and downstream conversion (for sales or support flows, did the interaction achieve its goal?). At least one of these signals should be computed and tracked as a KPI alongside the infrastructure metrics.

SLOs Appropriate for LLM Systems

Service Level Objectives for LLM systems require different calibration than traditional APIs, because quality degrades gradually and subjectively rather than switching between "up" and "down."

A reasonable starting framework for a customer-facing LLM feature:

• Latency SLO: P95 ≤ 3 seconds end-to-end (including upstream preprocessing). P99 ≤ 8 seconds. Streaming responses have a different SLO: first-token latency P95 ≤ 1 second.
• Availability SLO: API success rate ≥ 99.5% over a 30-day rolling window (allowing for vendor downtime and controlled graceful degradation).
• Format compliance SLO: ≥ 98% of responses match the expected output format. Format violations trigger a retry; if retry also fails, the fallback response is served and the event is counted against this SLO.
• Quality SLO: Weekly quality score (as measured by your judge metric) must remain within 10% of the 30-day baseline. A breach triggers an investigation; a breach for two consecutive weeks triggers a rollback evaluation.

The LLM Gateway as Observability Hub

The most effective observability architecture routes all LLM traffic through a single gateway layer rather than having application code call the LLM API directly. The gateway intercepts every request and response, computing and logging all five signals in one place without requiring each calling service to implement its own instrumentation.

A production LLM gateway does several things simultaneously: it handles retry logic and circuit breaking, it logs every request/response pair with a unique trace ID, it computes token usage, it applies output format validation, it enriches requests with metadata (user segment, feature flag state, model version), and it emits all signals to your observability platform. With a gateway in place, you can add a new metric across your entire LLM estate by changing one service rather than ten.

Alerting Thresholds and the Golden Signals Dashboard

The golden signals dashboard for an LLM system should show, in a single view:

• P50/P95/P99 latency over the last 24 hours vs. the 7-day baseline
• Error rate (by category) over the last hour vs. the 24-hour baseline
• Token usage per request over the last 24 hours vs. the 7-day baseline
• Format compliance rate over the last 24 hours
• Quality score over the last 7 days (quality signals are typically too noisy to chart hourly)
• User satisfaction proxy over the last 24 hours

Detecting Silent Quality Degradation

The hardest failure to catch is the one where nothing breaks - the system continues to return responses, latency is normal, error rate is zero - but the quality of those responses has degraded. This happens most often after a vendor model update.

The detection strategy requires a combination of approaches. Automated evaluation runs a judge LLM over a sample of recent responses and compares quality scores to a rolling baseline. Behavioral signals track the user satisfaction proxies described above. A/B holdouts keep a small percentage of traffic (1-5%) running against a pinned "known good" configuration so that you always have a current baseline to compare against.

The key discipline is to treat these quality checks as production alerts, not just dashboards. A quality degradation that is visible on a dashboard but produces no alert will sit unaddressed for days. Build the quality alert before you need it, not after.