Intermediate

Specs as Programs

A specification file is not documentation about what an agent does - it is the program the agent runs. Writing specs with that distinction in mind changes everything about how you write them.

AI School Editorial Team · Lilly Tech Systems Published Jun 25, 2026 · Reviewed Jun 25, 2026

The Central Insight

Traditional software is written in programming languages precisely because natural language is ambiguous. When you write if x > 0: return True, there is no question what the code does. When you write "check if the value is positive," there is plenty of room for interpretation: does zero count? What about NaN? What about a positive string?

AI agents do not execute code - they interpret instructions. This means that a natural-language specification file is, effectively, the program the agent runs. The quality of the output depends almost entirely on the quality of the spec. And unlike traditional code, a poorly written spec does not fail immediately with a clear error. It fails subtly, producing outputs that look plausible but miss the intent by just enough to cause problems.

The practical consequence: every hour you invest in spec quality pays off in reduced review cycles, fewer rollbacks, and agent outputs you can trust. The agents we run today have specs that were rewritten multiple times based on their failure modes. The rewriting is worth it.

The Spec-vs.-Runbook Distinction

Our fleet has two types of documents for each agent: a spec and a runbook. Understanding the difference is the first step to writing better specs.

Document	Answers	Audience	Changes When
Spec (e.g., `CONTENT-SPEC.md`)	What, when, and what not to do	The agent, in every run	The agent’s mission changes
Runbook (e.g., `ContentCreation-runbook.md`)	How to do it - the detailed method	Humans + the agent	The method improves

The spec is short, declarative, and unambiguous. It tells the agent what success looks like, what is out of scope, and what to do if it encounters edge cases. The runbook is longer and methodological. It contains the quality bar, the phase-by-phase procedure, the lessons learned from past runs, and the locked decisions that should not be relitigated. Both files are read by the agent at the start of every run.

The Five Sections Every Spec Needs

After iterating on specs for five different agents, a common structure emerged that reduces ambiguity without making specs unwieldy:

Trigger (when does the agent run?) - The schedule or event that starts the agent. Explicit: "Thursday ~7 PM ET" or "when a TesterBot report appears." Agents that run on ambiguous triggers tend to run at the wrong time or not at all.

Scope (what does the agent work on?) - The exact files, queues, or data sources the agent reads from and writes to. If a file is not in scope, the agent should not touch it. Name the files explicitly.

Decision rules (how does the agent choose?) - When there are multiple valid inputs, how should the agent pick? Priority ordering, tiebreakers, and skip conditions. The ContentCreation spec says: pick the oldest highest-priority unchecked backlog item, prefer items adjacent to high-engagement courses per the metrics file, skip items already in-progress or with an open PR.

Output contract (what must the output include?) - The exact deliverable the agent must produce, down to file names, required sections, and quality criteria. Vague outputs produce vague results. "Build an 8-lesson HTML course following the gold standard at ai-school/token-optimization/" is better than "build a course."

Explicit constraints (what must NOT happen?) - The most important section. Named prohibitions prevent the most costly failures. "NEVER push to master." "Do NOT generate promo images." "Do NOT modify unrelated files." "If the backlog is empty, open no PR." Constraints are the safety net.

Writing Constraints That Work

The constraints section deserves special attention because it is where most spec failures originate. The common mistake is writing constraints that describe intent rather than behavior: "be careful not to push to master" is weaker than "NEVER push to master - master auto-deploys to the live production site." The second version names the consequence, which gives the agent both the rule and the reason to follow it.

⚠

The asymmetry of constraint quality: A missing positive instruction causes the agent to under-produce - it skips something it should do, which you notice in review. A missing constraint allows the agent to over-produce - it modifies something it should not, which you may not notice until the change is live. Write constraints first; instructions second.

Constraints also need to anticipate the edge cases that are specific to your agent. ContentCreation-Draft’s constraints include: skip items that already have an open PR (prevents duplicate PRs), skip the item marked "manual - not for the routine" (prevents the agent from doing work reserved for the human), and do not generate promo images (the tools to do this are not available in the cloud environment). Each constraint was added after an edge case appeared in practice.

The Spec Is Versioned in Git

Keeping specs in the repository alongside the code they govern has three benefits that are not obvious until you need them:

Change history. Every spec change is a commit. When an agent starts producing different output, you can diff the spec against the previous version and see exactly what changed. This is invaluable for debugging regressions.
Co-evolution. When the thing the agent works on changes (new file structure, new quality bar, new site touchpoint to wire), the spec changes in the same PR. The agent’s behavior and the context it operates in stay synchronized.
Review trail. Spec changes go through the same PR process as code changes. A spec that would cause harmful agent behavior can be caught in review before it ships, just as bad code can.

Worked Example: Reading the ContentCreation Spec

The spec this very course was built from is at contentcreation/CONTENT-SPEC.md. Let us trace through its structure as a worked example of the five-section pattern:

Trigger: "Runs weekly, Thursday ~7 PM ET" - explicit schedule.
Scope: Reads TODO.md, METRICS.md, and the gold-standard course at ai-school/token-optimization/. Writes to a new course directory at ai-school/<slug>/. Also wires ai-school/index.html, main.js, sitemap.xml.
Decision rules: Pick ONE item, highest priority oldest first. Prefer items adjacent to high-engagement courses per METRICS.md. Skip in-progress or done items. Skip items with an open PR.
Output contract: 8-lesson HTML course. index.html + 8 lesson pages. Canonical/OG/Twitter/schema on every page. Byline row on every lesson. ≥1 lesson with outbound references. Wired into all five site touchpoints. PR titled "ContentCreation: <Course Name> (draft for review)".
Constraints: NEVER push to master. Do NOT merge the PR. Do NOT generate promo images. Do NOT modify unrelated files. If backlog is empty, open no PR.

📚

See also: The output contract in the ContentCreation spec references a quality bar from the runbook. For the prompt-level patterns that make agent instructions reliable, see Prompt Patterns That Survive Production - particularly the output-contracts and role-anchoring lessons.

The Spec Iteration Cycle

No spec is right on the first attempt. The iteration cycle for each agent looked like this: write the initial spec, run the agent, review the PR, note what was wrong or missing, update the spec, repeat. For ContentCreation-Draft, early iterations produced courses that were missing the autoMarkProgress branch in main.js, did not include the sitemap entries, or had relative links that did not resolve. Each failure produced a spec addition.

The implication: budget for two to three iteration cycles before an agent’s output consistently passes review without revision requests. The iterations are faster than writing the courses manually, but they are not free. The payoff is that once the spec stabilizes, the agent runs reliably for months without further adjustment - until the context it operates in changes and the spec needs updating again.