The Agent Frameworks Playbook

The whole course as an actionable playbook - a framework maturity model, the production-readiness checklist, the observability minimum bar, and ten rules that apply regardless of which framework you chose.

The Agent Framework Maturity Model

The Production Readiness Checklist

Before deploying any agent to production, verify:

Safety & Correctness

• ☐ Every tool has input validation - the agent cannot pass malformed input to a tool that causes unintended side effects
• ☐ Tool errors are handled explicitly, not silently swallowed - the agent sees a meaningful error message and can react
• ☐ There is a hard loop limit (iteration cap or token budget) that stops runaway agent loops
• ☐ Human-in-the-loop gates are in place for any action that is irreversible (sends email, modifies a database, calls a paid external API)

Observability

• ☐ Every run produces a structured trace with: input, tool calls (name + args + result), model responses, and final output
• ☐ Trace storage is configured - traces survive process restarts and are queryable by run ID
• ☐ You can reproduce any run by replaying its trace inputs
• ☐ Token usage is tracked per run (critical for cost attribution and budget enforcement)

Reliability

• ☐ The agent has been tested against tool failure scenarios (what happens if web_search returns an error?)
• ☐ Retry logic is configured for transient failures (network errors, rate limits) with backoff
• ☐ The agent degrades gracefully when it cannot complete - it returns a partial result or a clear failure message rather than hanging or crashing
• ☐ You have tested the agent on adversarial or malformed inputs

Operations

• ☐ Framework version is pinned in your dependency file
• ☐ You have a test suite for the agent’s core behaviors (not just “does it run”)
• ☐ There is a deployment runbook: how to roll back if the agent starts misbehaving in production

Observability Minimum Bar

The minimum viable observability stack for a production agent is three things:

1. Structured trace per run. At minimum: run ID, timestamp, input, list of tool calls with args/results, final output, error (if any). This is what you hand to the on-call engineer when they get paged at 2 AM.
2. Token count per run. Without this, you discover cost problems on the monthly bill rather than in the weekly review. Both LangGraph (via LangSmith) and the OpenAI Agents SDK emit token usage automatically. For CrewAI, access it via result.token_usage.
3. Error rate and latency dashboard. Even a simple time-series plot of “error rate by agent” and “p95 latency by agent” catches regressions within hours rather than days.

Anti-Patterns to Avoid

• The perfect framework fallacy. No framework is right for every use case. If you spend three weeks evaluating frameworks before writing a single tool, you are optimizing the wrong variable. Pick one, build the first agent, and migrate if the framework actively hinders you.
• Skipping traces for “simple” agents. Simple agents become complex when they fail unexpectedly. The agent you didn’t trace is the agent you cannot debug. Wire tracing into every agent, regardless of perceived complexity.
• Building without a loop limit. Every agent without a circuit breaker has an incident waiting to happen. Set the limit conservatively (10-20 iterations for most tasks) and increase it only when you have evidence the task requires more.
• Using the most capable model everywhere. Frontier models cost 10-50× more than capable mid-tier models. Most agent subtasks (tool result summarization, extraction, routing) do not require frontier capability. Route by task complexity - see the Token Optimization course for the full pattern.
• Ignoring framework major versions. Frameworks ship breaking changes in major versions. Pin your version, read the migration guide before upgrading, and test upgrades against your agent behavior before production deploy. A framework upgrade that silently changes agent behavior is one of the hardest production bugs to diagnose.

Ten Rules to Remember

1. Every agent needs a loop limit - add it before you add anything else.
2. Trace everything, from the first prototype. Tracing is cheapest to add at the start.
3. Tools should be narrow: one purpose, documented inputs, structured error returns.
4. The framework that matches your team’s mental model ships faster than the framework that scores highest on benchmarks.
5. Human-in-the-loop is not a last resort - it is an engineering requirement for any irreversible agent action.
6. Never use a frontier model for a task a cheaper model can handle reliably.
7. Framework abstractions that hide failures are worse than no framework at all.
8. Test agent behavior, not just agent execution. “It ran without crashing” is not a test.
9. Pin your framework version. The agent that works today may not work tomorrow after an upgrade.
10. The best framework is the one you ship, not the one you evaluate.

Where to Go Next