Advanced

LLM Failure Modes in Production

Before you can defend against LLM failures, you must name them. This lesson gives you the six-category taxonomy that covers every class of LLM production incident - with detection and prevention for each.

A Taxonomy of LLM Production Failures

LLM production failures don’t fit neatly into the failure categories that traditional software engineering developed. You can’t characterize them purely by error codes, stack traces, or service outages. Some of the most damaging LLM failures leave no error trace at all - they are quality failures that only appear in the content of responses.

The taxonomy below organizes LLM failures into six categories. For each category, we describe what it looks like from the outside (in logs and user reports), its root cause, how to prevent it, and how to detect it before it becomes a major incident.

Failure 1: Hallucination at Scale

What it looks like in logs: HTTP 200s everywhere. Format validation passes. Latency is normal. Users start submitting support tickets, complaints, or corrections. Internal reviewers notice factual errors. In some cases, a downstream system that relies on LLM output starts producing incorrect results.

Root cause: LLMs are generative models; they predict plausible tokens, not verified facts. At small scales, hallucinations are caught by manual review. At production scale, they become a statistical certainty - a model that hallucinates on 0.5% of queries is wrong thousands of times per day at meaningful traffic levels. The risk is compounded when the hallucinations pass format checks: a confidence score, a product name, a date, or a policy reference that looks correct but isn’t.

Prevention: Ground the model in retrieved context (RAG) for factual queries. Add explicit instructions not to guess or speculate outside of provided context. For high-stakes domains, implement answer verification as a second model call or deterministic lookup. Limit the model’s scope to tasks where hallucination risk is acceptable.

Detection: A judge LLM that scores factual accuracy over sampled outputs. Consistency checks (ask the model the same question twice; inconsistent answers signal hallucination risk). User feedback signals (explicit thumbs-down, regeneration requests, follow-up corrections). For RAG systems, citation checks (does the answer appear in the retrieved context?).

Failure 2: Prompt Injection

What it looks like in logs: Normal request logs. The only signal is in the output content: the model ignores its system prompt and follows instructions that came from user input or retrieved documents. Symptoms include responses that reveal the system prompt, take actions outside the system’s intended scope, or adopt a persona the operator did not authorize.

Root cause: LLMs do not have a hardcoded distinction between "instructions" and "data." When user-provided content is included in a prompt, the model may follow instructions embedded in that content as if they were authoritative. This is especially dangerous when user content is passed through retrieval (documents the user doesn’t control may contain injections) or when the system uses the LLM to process external data.

Prevention: Use structured prompt templates that clearly delineate user content from system instructions. Apply input sanitization that strips instruction-like patterns from user inputs. Never reveal the system prompt to users, and instruct the model explicitly not to reveal it. For agentic systems, apply the principle of least privilege: the model should not have access to capabilities it doesn’t need.

Detection: Automated red-team inputs in your CI gate (10+ injection attempts, all expected to be rejected). Output monitoring that flags responses that contain system prompt content or that invoke out-of-scope capabilities. Manual red-team exercises quarterly.

Failure 3: Context Overflow

What it looks like in logs: Truncation may or may not be visible depending on the vendor API. If the vendor silently truncates, the model call succeeds but the model has seen a different (shorter) input than you intended. The model’s answers become inconsistent, seem to "forget" earlier conversation turns, or give answers that are correct for a different context than the one you thought you provided.

Root cause: Every LLM has a maximum context window. When the total prompt (system message + conversation history + retrieved context + user input) exceeds that window, the provider either truncates or returns an error. Truncation is the more dangerous outcome: the model processes a modified version of your prompt without any visible failure signal.

Prevention: Count tokens explicitly before sending every prompt. Enforce a hard token budget for each component of the prompt (system: X tokens, history: Y tokens, context: Z tokens, user input: N tokens) and trim or summarize components that exceed their budget. Never rely on the model to handle overflow gracefully.

Detection: Log prompt token counts for every call. Alert when any prompt exceeds 85% of the context window limit. Track the rate of calls that hit the max_tokens output limit (a signal that the input may be consuming most of the available context).

Failure 4: Rate Limits and Quota Exhaustion

What it looks like in logs: HTTP 429 errors from the vendor API, often beginning at a specific time (peak usage, a batch job, a traffic spike). The error rate spikes. If no circuit breaker is in place, requests queue up and latency blows out. In the worst case, the system enters a retry storm: every failed request retries, increasing load on the rate-limited API, which increases failures, which increases retries.

Root cause: Vendor LLM APIs have per-minute and per-day rate limits on requests, tokens, or both. These limits are enforced at the account or key level, so a batch job and a user-facing application sharing the same key compete for the same quota. Unexpected traffic spikes, runaway agent loops, or a large batch job can exhaust quota that user-facing traffic needs.

Prevention: Separate API keys (and ideally separate accounts or projects) for different workloads with different priority levels. Implement a circuit breaker that stops sending requests when the rate limit is being hit, rather than continuing to retry. Use batch APIs for non-urgent work so they consume quota on a different tier. Monitor quota consumption and alert before exhaustion.

Detection: Track 429 error rate per API key and per workload. Alert when 429s appear at all for user-facing keys. Track daily token consumption against quota and alert at 70% consumed by hour 18 of the day.

Failure 5: Latency Spikes (P99 Blowout)

What it looks like in logs: P50 and P95 latency look normal. P99 latency is 5-10× higher. Users experiencing the worst 1% of requests see timeouts, incomplete streaming responses, or UI freezes. The P99 tail may only appear under load or when specific input patterns (very long inputs, complex multi-step reasoning requests) trigger longer generation.

Root cause: LLM latency scales with output length, input length, and system load at the vendor. A request that generates a very long response, or that arrives when the vendor’s systems are under load, takes dramatically longer than a typical request. At high concurrency, these slow requests hold connections and contribute to cascading latency increases.

Prevention: Set a max_tokens limit that reflects the maximum acceptable response length for your use case, not the vendor maximum. Use streaming responses so that users see partial output rather than waiting for full generation. Implement a per-request timeout that aborts if a response hasn’t started arriving within the first-token latency SLO.

Detection: Track P99 latency alongside P50 and P95. Alert separately on P99. Track first-token latency for streaming responses as a leading indicator of system load. Correlate P99 spikes with input length distribution to identify input-driven latency patterns.

Failure 6: Silent Quality Drift

What it looks like in logs: Nothing. Error rate is normal. Latency is normal. Format compliance is normal. Months later, someone notices that the chatbot’s answers have changed in tone, or that the summarizer is missing details it used to include, or that the code generator is producing subtly different patterns. Investigation reveals the change was gradual and began around the time of a vendor model update.

Root cause: LLM vendors update their models continuously. Fine-tuning, safety training, and capability improvements change model behavior in ways that are not always announced or documented in granular detail. If you are not pinning to a specific model version, or if the pinned version itself receives an update, behavior can shift without any deployment on your end.

Prevention: Pin to specific model versions wherever the vendor allows it. Maintain a "known-good" prompt and model configuration snapshot that can be used as a comparison baseline. Run weekly automated evaluations against a held-out test set so that behavioral changes surface as metric drops rather than user complaints.

Detection: Weekly quality evaluation pipeline comparing current model output against baseline. A/B holdout traffic against a pinned known-good configuration. User satisfaction trend monitoring with alerts on sustained drops.

Failure Summary Table

Failure Mode	Symptom	Detector	Prevention
Hallucination at scale	Wrong answers; user complaints; no errors	Judge LLM; citation check; user feedback	RAG grounding; scope limiting; verification layer
Prompt injection	Model ignores system prompt; out-of-scope actions	Red-team CI gate; output content scan	Input sanitization; structured templates; least privilege
Context overflow	Model "forgets" context; inconsistent answers	Token count logging; context budget enforcement	Explicit token budgets; prompt trimming; summarization
Rate limit exhaustion	429 errors; latency blowout; retry storm	429 rate per key; quota consumption alert	Key separation; circuit breaker; batch tier for non-urgent work
Latency spikes (P99)	Timeouts; incomplete responses at tail	P99 latency alert; first-token latency	max_tokens limit; streaming; per-request timeout
Silent quality drift	Subtle behavior change; no errors; user trend	Weekly eval pipeline; A/B holdout; satisfaction trend	Model version pinning; baseline snapshots; scheduled evals