Advanced

Monitoring and the Metrics Feedback Loop

An agent that runs and produces output is not the same as an agent that achieves its goal. The feedback loop between goal-level measurement and spec refinement is what makes a fleet improve over time rather than plateau.

AI School Editorial Team · Lilly Tech Systems Published Jun 25, 2026 · Reviewed Jun 25, 2026

Three Layers of Agent Metrics

Monitoring an AI agent fleet requires measuring at three distinct levels. Most teams start with the first level and stop there; the compounding value is in all three:

Layer	Question	How We Measure	What It Tells You
Execution	Did the agent run?	Cron schedule logs, GitHub Actions history	Infrastructure reliability; silent failure detection
Output quality	Was the output merged?	PR merge rate, revision request rate, PR iteration count	Spec quality; agent capability alignment with task
Goal achievement	Did the output achieve what we wanted?	LinkedIn analytics (CTR, impressions), Google Analytics (course views, engagement time)	Whether the fleet is actually moving the needle on the underlying objective

The execution layer is the easiest to instrument. If a cron job runs and the agent produces a PR, the execution metric is green. But a fleet that reliably executes and reliably produces PRs that get merged could still be failing at the goal level - if the courses are not driving traffic, or the posts are not driving engagement, the fleet is busy but not effective.

The Metrics Log: METRICS.md

Our goal-level metrics are logged in contentcreation/METRICS.md - one row per course, measured approximately one week after the corresponding LinkedIn post goes live. The columns capture the three goal-level measurements that matter most for the ContentCreation agent:

Post performance: Impressions, clicks, click-through rate (CTR), engagement rate. These measure whether the distribution mechanism (LinkedIn post) is working.
Course performance: Total views, unique users, average engagement time, deepest lesson visited. These measure whether the content itself is retaining readers who arrive.
Action taken: What we did in response to the signal. This closes the loop: measurement without action is data collection, not a feedback loop.

📚

Real numbers from our log: The Token Optimization course launched with a 19.32% CTR on 88 impressions - the visual and hook were working, but distribution was limited. The Prompt Patterns course launched with 1.61% CTR on 62 impressions - the distribution was similar but the visual did not stop the scroll. The Production-Checklist lesson in that course averaged 25 minutes 24 seconds engagement time, a massive standout versus the site average of ~9 seconds. Each signal drove a specific action recorded in the log. Illustrative benchmarks: a typical LinkedIn CTR is roughly 0.5-1%; >5% indicates strong visual/hook performance.

The Signal-to-Action Table

The most important design element of the monitoring system is not the metrics themselves but the action mapping: for each signal pattern, what do you do? Without pre-defined responses, metrics become a dashboard to feel informed rather than a system for driving improvement.

Signal	Diagnosis	Action
Post impressions <200 by day 7	Distribution failure, not content failure	Change post time; ask team to react in first hour (early engagement boosts reach); repost with different hook in 2-3 weeks
CTR <5%	Hook or visual not stopping the scroll	Design a new artifact concept for next course; rewrite the hook formula
Good clicks, users << clicks	Tracking loss or instant bounce	Check page load speed; accept ~30-40% LinkedIn in-app loss as normal floor
Index views but no lesson views	Course landing page not converting readers	Sharpen lesson-card copy; add a stronger Start Course CTA above the fold
Lesson avg engagement <1 min	Content not holding readers past the intro	Rewrite that lesson’s opening; add a worked example earlier
One lesson dramatically outperforms	That topic/format resonates strongly	Add a spin-off course idea to the backlog; use that lesson’s format as a template
Organic search arrivals	SEO working for that topic	Add more courses in that cluster; prioritize adjacent backlog items
High traffic + high engagement	Winner - this is working	Deepen the cluster; use as the benchmark for the next course’s quality bar

How Metrics Drive the Next Spec

The metrics feedback loop has a direct channel into the ContentCreation agent’s decision-making: the spec instructs the agent to prefer backlog topics adjacent to courses with high engagement, and deprioritize topics adjacent to courses with weak engagement. This means METRICS.md is not just a reporting artifact - it is an input to the agent’s pick logic on every run.

The practical effect: if a course on agent frameworks performs well (even after initial distribution problems), the agent should pick the next agent-adjacent topic in the backlog. If a compliance course underperforms consistently, agent-adjacent topics should be favored over compliance-adjacent ones. The fleet adapts its direction based on what works, rather than following a static plan that ignores market response.

The Measurement Infrastructure

Two external systems provide the goal-level data for our ContentCreation feedback loop. Neither requires custom instrumentation - both are standard tools used through their existing interfaces:

LinkedIn Analytics provides post performance data for the AI School showcase page. The key metrics are impressions (total post views), clicks (link clicks), CTR (clicks ÷ impressions), and engagement rate (total interactions ÷ impressions). These are available through the LinkedIn showcase admin panel at the page-level analytics view. For authoritative documentation on LinkedIn analytics metrics and definitions, see the LinkedIn Help: Understanding Page Analytics documentation.

Google Analytics 4 (GA4) provides the course usage data. The relevant report is Engagement → Pages and Screens, filtered by slug, over the launch-week date range. The key metrics are Views (total page loads), Users (unique visitors - the most reliable traffic indicator), Average engagement time (time actively on-screen, not just session duration), and deeper lesson penetration (the fraction of unique users who visit lesson pages beyond the index). For GA4 metric definitions and report guidance, see the Google Analytics Help: About Engagement reports. Note that GA4 engagement time measures active engagement (the tab is in focus and the user is interacting), making it a more reliable signal of genuine reading than older session-duration metrics.

⚠

Measurement caveats: Our GA property has no internal-traffic filter, so owner and agent verification visits inflate numbers - especially for courses with low traffic. Treat small counts as upper bounds. The LinkedIn in-app browser undercounts GA users by 30-40% because many users never leave the LinkedIn app. Accept this as the floor and compare relative performance across courses rather than comparing absolute numbers to external benchmarks.

The Compounding Loop

The mechanism that makes a fleet more valuable over time is the compounding loop: measurement reveals what works, what works drives backlog prioritization, better-prioritized backlogs produce higher-performing outputs, better performance produces better measurement data, and the cycle continues. Each week’s measurement improves the next week’s spec input.

This loop is slow at first - you need several data points before patterns emerge. But after three to four months of operation, the signal is clear enough that the agent’s pick logic is meaningfully guided by the metrics rather than just defaulting to the oldest item. The compounding effect is real: a fleet that measures systematically outperforms one that does not, even if both start with equivalent specs and agents.