Intermediate

Build Your Own Fleet: The Playbook

A fleet is not a one-day project. This lesson gives you the maturity model, the 30-day plan, the design checklist, and the patterns that make the fleet compound in value over time.

AI School Editorial Team · Lilly Tech Systems Published Jun 25, 2026 · Reviewed Jun 25, 2026

The Fleet Maturity Model

Fleet maturity evolves in stages. Each stage adds value but also adds complexity. Jumping from zero to the highest maturity level is a reliable way to fail. Start at Stage 0 and progress when the previous stage is stable:

Stage	Name	What You Have	Human Involvement
0	Fully Manual	A team doing repetitive work manually. No agents. No specs. Baseline.	100% - everything
1	Assisted	One agent for the most repetitive, best-understood task. Human-in-the-loop on every output. Spec in draft; iterating.	High - reviews every PR
2	Coordinated	Two to four agents. Shared blackboard (TODO file). Spec stable for each agent. PR gate + CI checks. METRICS.md started.	Medium - reviews PRs, manages backlog
3	Measured	Full fleet. Metrics feedback loop driving backlog prioritization. Failure taxonomy documented. Recovery procedures defined.	Low - strategic decisions, backlog curation, exception handling
4	Self-Improving	Metrics signals flow back into agent specs automatically. The fleet identifies its own gaps and proposes backlog items. Spec updates are themselves PR-gated.	Minimal - approves major decisions; monitors for anomalies

💡

Where we are: Our fleet operates at Stage 3. The metrics feedback loop is running and influencing backlog prioritization. Stage 4 - where metrics automatically generate spec updates - is on the roadmap but requires strong track record and audit infrastructure before it is safe to deploy. Do not rush to Stage 4.

Which Agent to Build First

The most common mistake when starting a fleet is trying to automate the most impressive or most ambitious task. Start instead with the task that has all four properties of a good first agent:

Fully repeatable. The same sequence of steps, every time. No exceptions, no judgment calls, no creative decisions.
Well-understood. You have done this task manually enough times to write a complete spec without guessing. If you are still figuring out the process yourself, the agent will amplify your confusion rather than encode your expertise.
Low blast radius. When (not if) the agent makes a mistake, the worst case is manageable. Do not start with an agent that can publish to your production site, send emails to customers, or make purchases.
Measurable output. You will know within hours or days whether the output was good. This tight feedback loop is essential for iterating the spec quickly.

The 30-Day Fleet Stand-Up Plan

This plan is for building Stage 1 maturity with one agent and progressing toward Stage 2. Adjust the timeline based on how much time you can dedicate per week:

Days 1-3

Task audit. List every recurring task in your workflow. For each one, score it on the four properties above (repeatability, understanding, blast radius, measurability). Identify the highest-scoring candidate. If no task scores well on all four, do not build an agent yet - document the task more thoroughly until you can write a complete spec for it.

Days 4-7

Write the spec. Draft the five-section spec (trigger, scope, decision rules, output contract, constraints). Hand the draft to a colleague and ask them to try following it. Where they ask clarifying questions, you have found spec gaps. Revise until no questions remain.

Days 8-10

Tool grant design. Map the minimal set of tools the agent needs. For each write tool, document the worst-case misuse. Remove any write tools where the worst case is unacceptable; replace with read-only alternatives or add hard constraints. Set up repository branch protections if you have not already.

Days 11-14

First agent run. Run the agent with a human observer. Do not intervene unless the agent is about to do something irreversible and harmful. Let it fail if it is going to fail - the failures reveal spec gaps. Document everything the agent did that surprised you.

Days 15-21

Spec iteration. For each surprising behavior from the first run, update the spec. Add constraints, clarify decision rules, or sharpen the output contract. Run the agent again. Repeat until the output consistently passes review without revision requests. This typically takes two to four iterations.

Days 22-28

Stabilize and add monitoring. Once the agent is running reliably, add execution monitoring (cron schedule confirmation), output quality tracking (PR merge rate, revision rate), and start measuring goal-level outcomes if applicable. Add the task to the TODO blackboard so future work items for this agent flow through the shared queue.

Days 29-30

Document the failure taxonomy. Based on the iteration runs, write down the failure modes you observed and the recovery procedure for each. This documentation is the foundation for Stage 2 - when you add the second agent, you will need this for training the reviewers and for diagnosing problems without having to reconstruct what happened.

The Fleet Design Checklist

Before declaring a new agent ready for production, verify each item in this checklist:

✅

Fleet Agent Readiness Checklist

Spec covers all five sections: trigger, scope, decision rules, output contract, constraints
Every constraint has been tested against a real or plausible edge case
Tool grants follow least-privilege: minimum writes needed, no merge or delete access
Branch naming convention defined; branch protection rules enforcing the PR gate
CI checks defined for the output (link check, HTML validation, or equivalent)
Failure taxonomy documented: at least two failure modes with recovery procedures
PR review checklist defined: what does a reviewer actually check in this PR?
Goal-level metrics defined: how will you know if this agent is achieving its purpose?
Backlog hygiene defined: who adds items, what format, what constitutes a complete item?
Agent has run at least three times with output reviewed and no unexpected behaviors in the last two runs

The Compounding Advantage

The reason to invest in a fleet rather than a single agent is the compounding dynamic that emerges across multiple agents sharing a coordinated workflow. Each agent compounds along a different axis:

Spec quality compounds. The lessons learned writing the first spec make the second spec better from day one. By the fifth agent, the spec template is so refined that first-run error rates are dramatically lower than they were for agent one.
Metrics compound. Each week of measurement adds a row to the metrics log. After six months, you have enough data to see seasonal patterns, topic clustering effects, and format performance trends that are invisible in one or two weeks of data.
Backlog quality compounds. As the metrics feedback loop runs, backlog items are increasingly well-chosen - topics that are likely to perform well based on observed evidence rather than guesses. The fleet starts producing better work not because the agents got smarter but because they are being pointed at better problems.
Recovery speed compounds. The failure taxonomy grows with each incident. Recovery from a new failure class takes hours. Recovery from a previously-documented failure class takes minutes because the playbook already exists.

Common Mistakes and How to Avoid Them

Mistake	Why It Happens	Prevention
Building too many agents too fast	Enthusiasm after first agent succeeds	Fully stabilize each agent (3+ runs, no unexpected behaviors) before adding the next
Skipping the spec review step	Impatience to see the agent run	The spec review (hand-it-to-a-colleague test) is not optional. It is cheaper than two failed agent runs.
Using metrics as a report, not a feedback loop	Measuring without defining the action table	Write the signal-to-action table before collecting the first data point
Removing the human gate too early	Trust in the agent after a run of good PRs	Define explicit graduation criteria: N consecutive PRs with zero revision requests, operating for at least M weeks
Not documenting failures	Urgency of recovery; deferred logging	Write the postmortem within 24 hours while the incident is fresh; add to the failure taxonomy immediately
Confusing agent failure with spec failure	Attributing all errors to the model	Read the agent’s reasoning trace (if available) or re-run with the same spec to distinguish a spec ambiguity from a model error

The Ten Rules of Fleet Operation

Specs are programs. Write them with the same rigor you apply to code.
Constraints come first. Write what the agent must NOT do before writing what it should.
Git is your coordination layer. Do not add infrastructure until git-based coordination breaks down.
PRs are non-negotiable gates. Nothing a fleet agent produces goes live without human review - at least until stage 3 maturity is proven and the risk profile justifies automation.
Measure at the goal level. Agent execution is an input. Goal achievement is the output. Measure the output.
Document every failure. The failure taxonomy is the most valuable document in the fleet. Invest in it.
One agent at a time. Stability first, scale second. A stable single-agent pipeline is more valuable than three unstable ones.
Backlog hygiene is human work. Agents execute; humans prioritize. Do not automate backlog curation until Stage 4 and only with strong oversight.
The compounding loop needs data. Measure every week. Act on signals. The value does not arrive until the data accumulates.
Build for recovery. Every agent action must be reversible. If it cannot be reversed, add a human gate before it.

📚

Course summary: You have covered the full fleet pattern: the architecture (Lesson 1), the spec-as-program model (Lesson 2), the blackboard coordination pattern (Lesson 3), tool design (Lesson 4), guardrails (Lesson 5), metrics feedback loops (Lesson 6), failure recovery (Lesson 7), and this build playbook (Lesson 8). For the code-level view of agent frameworks this fleet builds on, see AI Agent Frameworks in Practice. For production operations, see Production Readiness Runbook for LLM Systems.