Failure Modes and Recovery

Every production AI fleet fails in characteristic ways. Knowing the failure taxonomy and the matching recovery patterns is the difference between a recoverable incident and a production crisis.

A Taxonomy of Fleet Failures

Fleet failures do not all look the same. After months of operating this setup, four distinct failure classes emerged, each with different causes, different signatures, and different recovery paths:

The Zombie Deploy Incident

The most memorable failure in our fleet’s history is what we call the zombie deploy. In early operation, the GoDaddy FTPS deploy action was triggered by every push to master. This meant that rapid consecutive pushes - a human fixing something right after an agent merged, for instance - could queue up multiple deploy jobs. Each new push superseded the previous queued deploy in the concurrency group, meaning earlier jobs were cancelled before they completed their file uploads.

The zombie problem: a cancelled deploy does not roll back changes it had already uploaded. It just stops uploading the remaining files. This left the production site in a partially-deployed state where some files reflected the newer commit and others still reflected the older state. The result was broken cross-links: pages referencing files that had been renamed in the newer commit but not yet deployed.

The incident lasted approximately 4.5 hours before it was diagnosed and a full-site re-deploy resolved it. The fix had two parts: documenting the "batch pushes" risk in the runbook (to prevent rapid consecutive pushes) and adding a manual full-deploy trigger to the GitHub Actions workflow for recovery situations. The spec for ContentCreation-Draft now explicitly notes this risk and instructs the agent to push once per session.

The Committed Draft Incident

The second notable failure was the committed draft: early in the ContentCreation pipeline, the runbook described a two-phase workflow - draft the content as Markdown, get human review, then build the HTML. A cloud agent session that did not understand the local-vs.-cloud distinction followed the runbook literally: it wrote the Markdown draft to the repository, committed it, and opened a PR that included the draft file.

The Markdown file passed review without the reviewer noticing it was a draft artifact. When the PR merged, the file deployed to the live site as publicly-accessible content - an unformatted draft that was visible to anyone who found the URL directly. It was removed within minutes of discovery, but the incident established a critical rule: draft content must never be committed to the repository. The runbook now states explicitly: "put the draft OUTSIDE the deployed tree or delete it before pushing." The spec instructs the cloud agent to build HTML directly without an intermediate draft step.

Hallucinated Link Failures

A subtler and more frequent failure mode is the hallucinated internal link. When building course HTML, the agent must create relative links to companion courses, images, and other assets. In some cases, the agent links to paths that do not exist in the repository - it knows a course on a topic should exist based on its training knowledge or the catalog structure, but that specific course has not been built yet.

The failure signature: the HTML is syntactically valid and passes basic review, but clicking certain links produces 404 errors on the live site. The spec now includes an explicit self-check step: "verify every relative href in the new pages resolves to a real file in the checkout." The agent runs this check and removes or adjusts links to files that do not exist. When links to non-existent courses are removed, they are candidates for the backlog - the hallucinated link often identifies a real gap in the catalog.

Scope Creep: When Agents Wander

Scope creep failures occur when an agent modifies files outside its assignment. The typical pattern: the agent notices something "wrong" with an existing page while reading it for context, and decides to fix it even though fixing it is not in the spec. The fix may or may not be correct, but either way it represents an unauthorized change that was not requested and was not reviewed with that intent in mind.

Scope creep is insidious because the agent’s reasoning is often locally sensible. It sees a broken link or a missing alt attribute and corrects it - helpful behavior in a different context, but wrong behavior for a specialized fleet agent. The remediation is at the spec level: "Do NOT modify unrelated files. If you notice issues with existing pages, note them in the PR description instead of fixing them." The PR description becomes the escape valve - the agent can flag what it noticed without acting on it.

Recovery Patterns

For each failure class, a matching recovery pattern exists. The good news: because agents work through PRs on branches, most failures are recoverable with git operations that take minutes:

Prevention Is Cheaper Than Recovery

The recurring theme across all four failure classes: prevention through better specs and tighter tool grants is much cheaper than recovery after a failure. Each incident above added one or two lines to the spec or runbook. Those additions cost minutes to write. The incidents they prevent cost hours to recover from and sometimes affect real users on a live site. The failure taxonomy is worth maintaining precisely because it tells you where to invest prevention effort.