The Pre-Deploy Checklist

A 40+ point checklist that every LLM system should clear before it ships. Run it in CI. Run it before every significant prompt or model change. Run it because the alternative is finding out what you missed in production.

How to Use This Checklist

This checklist is organized into seven sections that map to the seven categories of LLM production failure. Items marked (Critical) are the ones that, if skipped, produce the highest-severity incidents. They should be treated as hard blockers - a deploy cannot proceed until they are checked. All other items are strong recommendations that should only be waived with documented reasoning.

The checklist is designed to be run against any system where LLM calls are part of a user-facing or business-critical path. For internal tooling and lower-stakes applications, a lighter version covering only Critical items is acceptable; for externally-facing, revenue-critical, or regulated systems, every item applies.

Section 1: Infrastructure Readiness

1. ☐ (Critical) API keys and credentials are stored in secrets management (not hardcoded or in environment files that are committed).
2. ☐ (Critical) Rate limit headroom is confirmed: current production throughput target is below 70% of vendor rate limits, with a documented burst strategy for the remainder.
3. ☐ Retry logic is implemented with exponential backoff for all LLM API calls, with a maximum retry count and a final fallback behavior.
4. ☐ Timeout configuration is set for all LLM calls; timeouts are tuned to P95 latency plus 2× headroom, not the vendor-documented maximum.
5. ☐ (Critical) A circuit breaker is in place: if the LLM API returns errors above a threshold rate, the circuit opens and requests are served from a fallback (cached response, graceful degradation, or error message) rather than piling up.
6. ☐ Load testing has been performed at 2× expected peak; latency P95 remains within SLO at that load level.
7. ☐ Infrastructure costs at projected production scale have been estimated; cost alerts are configured at 80% and 100% of budget. (Note: pricing figures here are illustrative - check current vendor pricing.)
8. ☐ Multi-region or multi-provider failover is documented; if the primary LLM provider is unavailable, the team knows within 5 minutes and has a runbook for switching.

Section 2: Prompt & Model Configuration

1. ☐ (Critical) System prompt is version-controlled alongside application code; the deployed version is traceable to a specific commit or tag.
2. ☐ (Critical) Model version is pinned where the vendor supports it; if pinning is not available, the team monitors for behavioral changes after vendor updates.
3. ☐ Temperature, top-p, max tokens, and other sampling parameters are explicitly set and documented; defaults are not relied upon.
4. ☐ The prompt has been tested against inputs at the maximum expected length; context window limits are documented and enforced upstream.
5. ☐ Input sanitization is in place: user inputs are scrubbed or escaped before insertion into prompts to prevent prompt injection.
6. ☐ The prompt has been tested with non-English inputs and with inputs that include special characters, code, and structured data formats.
7. ☐ A labeled test set of at least 100 representative inputs exists; the prompt passes at least 90% on quality criteria against this set.
8. ☐ The prompt behavior has been validated after any recent model update or parameter change.

Section 3: Output Validation

1. ☐ (Critical) Output format is validated programmatically for every response: if the system expects JSON, JSON is verified; if structured text is expected, a parser confirms the structure.
2. ☐ (Critical) A fallback is defined for invalid output format: the system does not pass malformed LLM output downstream; it retries, truncates gracefully, or returns an error.
3. ☐ Output length limits are enforced: responses that are truncated at the max_tokens boundary are detected and handled rather than silently delivered.
4. ☐ Sensitive data is not echoed back in outputs: a PII/secrets scan is run on a sample of outputs in staging.
5. ☐ At least one automated quality metric is computed per response (e.g., format compliance, similarity to ground-truth answers, refusal rate); this metric is logged.
6. ☐ The system has been tested for the most common hallucination pattern in its domain; a mitigation is in place (e.g., grounding via RAG, output fact-checking, user-visible confidence indicators).

Section 4: Testing & Red-Teaming

1. ☐ (Critical) A regression test suite exists covering the 20 most important user scenarios; it runs in CI on every code change and on a schedule to catch model drift.
2. ☐ At least 10 adversarial inputs have been tested: prompt injection attempts, jailbreak patterns, and inputs designed to elicit harmful or off-policy outputs.
3. ☐ Edge-case inputs have been tested: empty string, maximum-length string, inputs in unexpected languages, inputs with only special characters.
4. ☐ The system has been tested with simulated rate limit responses and API timeouts to verify fallback behavior.
5. ☐ Outputs have been reviewed by at least one domain expert who is not the engineer who wrote the prompt.
6. ☐ A "chaos day" or equivalent exercise has been run: dependencies were deliberately failed to confirm that the system degrades gracefully.
7. ☐ Red-team results are documented; any findings that are not resolved have an accepted-risk sign-off from a responsible party.

Section 5: Monitoring & Alerting

1. ☐ (Critical) Latency (P50, P95, P99) is instrumented and dashboarded for all LLM calls.
2. ☐ (Critical) Error rate (API errors, format violations, fallback triggers) is instrumented and has an alert threshold.
3. ☐ Token usage per request (input and output) is logged; a budget alert fires if average token usage exceeds expected by >50%.
4. ☐ A quality signal (hallucination rate, format compliance rate, or user satisfaction proxy) is computed and logged per session or per day.
5. ☐ An alert is configured to fire if the quality signal degrades by >10% week-over-week without a corresponding code change.
6. ☐ A dashboard exists that shows all five signals in one view; it is the first thing the on-call engineer checks during an incident.

Section 6: Security

1. ☐ (Critical) System prompt content is not exposed to end users; there is no prompt disclosure path via adversarial inputs.
2. ☐ User inputs are logged for audit purposes but are subject to data retention and privacy policies (GDPR, CCPA, or equivalent as applicable).
3. ☐ The LLM is not given tools or system access beyond what is strictly required for its function (principle of least privilege).
4. ☐ Outputs that include URLs, code, or executable content are sanitized before rendering in the user interface.
5. ☐ An access control review has confirmed that the LLM cannot be used to exfiltrate data beyond the user's own permissions.

Section 7: Rollback Plan

1. ☐ (Critical) The current production prompt and model configuration are snapshotted in version control; the team can deploy the previous version within 15 minutes.
2. ☐ Rollback criteria are defined in writing: what specific metric degradation triggers a rollback, and who has authority to initiate it.
3. ☐ The rollback procedure has been tested in staging; the team has performed a drill within the last 90 days.
4. ☐ If the LLM component cannot be rolled back independently of the application (e.g., because the API changed), a feature flag exists that disables the LLM path and enables a non-LLM fallback.

Readiness Tiers

Not every deployment needs the same level of hardening. The table below summarizes what each tier requires. Use the minimum viable tier only for internal tooling; anything user-facing should target at least Production Hardened.