Mapping Token Waste

Token waste is not random. It concentrates in predictable patterns that appear in almost every AI application. Learn to recognize the seven patterns, measure your baseline, and build the habit of profiling before optimizing.

Why Waste Hides in Plain Sight

Engineers who read their prompts often believe they are clean. The system instruction is short. The user message is brief. The retrieved context looks relevant. But prompts do not arrive at the model the way they look in source code - they arrive as assembled context windows where system instructions, few-shot examples, conversation history, retrieved documents, and tool results are all concatenated together. What looks like three small pieces becomes a 4,000-token context by the time it reaches the API, and most of those tokens were not intentional.

The first step in any optimization effort is measurement. Before you apply any technique, you need a baseline: tokens per request, tokens per session, cost per successful task. Without that baseline, you cannot know whether the levers you apply are working - or whether you are optimizing the wrong part of the stack.

The Seven Common Waste Patterns

1. Static System Prompt Repeat

Every request re-sends the full system prompt, including instructions, persona definitions, guardrails, and formatting rules. For applications with long system prompts (500-2,000 tokens is typical), this is a fixed overhead on every call. Across thousands of daily requests, it compounds into significant spend. Prompt caching eliminates most of this - but teams must profile to see how large the overhead is before deciding to invest in caching infrastructure.

2. Redundant History Resent

Chat applications typically append the full conversation history to every new message. A 10-turn conversation that started with a brief greeting will include all 10 exchanges when the user sends their 11th message. Without history truncation or summarization, token count grows linearly with conversation length. Long-running sessions can spend 70% of their token budget re-sending context the model already processed in previous turns.

3. Over-Retrieved RAG Context

Retrieval-augmented generation (RAG) systems pull documents or chunks from a vector store based on embedding similarity. A common misconfiguration is retrieving too many chunks (top-10 or top-20 is frequent) when top-3 or top-5 would cover the relevant content. Every extra chunk adds hundreds of tokens per request, dilutes the context with weakly-relevant material, and can actually reduce response quality by drowning the relevant signal.

4. Verbose Output That Is Then Discarded

Many pipelines generate outputs that are immediately parsed, summarized, or filtered downstream. A classification step that returns a 500-word explanation when all the pipeline needs is a label code. A tool-call response that includes full documentation when only the return value matters. Output tokens cost more than input tokens, and generating content that will not be used is pure waste.

5. Duplicate Content Across Turns

Agentic workflows often pass the same document, schema, or reference material to the model multiple times across a multi-step task. The model reads the same database schema in step 1 (to plan a query), step 3 (to validate output), and step 5 (to format results). Deduplicating context across steps - or using a shared context object instead of re-injecting - eliminates this pattern.

6. JSON or XML Schema Verbosity

Structured output prompts often include full JSON schemas, XML templates, or Pydantic model definitions in the prompt. These are frequently hundreds of tokens long. A schema that defines 15 fields with descriptions, type annotations, and validation rules can consume more tokens than the actual content being extracted. Compact schema representations, or moving schemas to a cached system prompt section, substantially reduces this overhead.

7. Formatting Filler in Instructions

Instructions that explain themselves at length ("Please be sure to provide a thorough and comprehensive response that addresses all aspects of the question in a clear and organized manner...") add tokens without adding precision. Tight, imperative instructions ("Answer in 3 bullet points. No preamble.") use a fraction of the tokens and typically produce better-controlled outputs.

Establishing Your Baseline

Before applying any optimization, measure these numbers for each prompt or pipeline in your application:

• Tokens per request: Input tokens, output tokens, and total. Log these for every call via your API client's usage field.
• Cost per request: Multiply by current model pricing. Even a rough calculation reveals which pipelines are driving spend.
• Request distribution: Not all requests are equal. Profile the 90th-percentile request as well as the average - occasional long requests often drive a disproportionate share of cost.
• Tokens per successful task completion: The best optimization metric. If you reduce tokens but quality drops and users retry more, total spend may increase.

Tools for Token Counting and Profiling

Most major model providers expose token counts in API responses. For pre-request estimation, tokenizer libraries allow you to count tokens before making a call:

• Anthropic: The Messages Count Tokens API returns an exact count without consuming API quota. Useful for profiling in staging before production deployment.
• OpenAI: The tiktoken library provides offline token counting for GPT model families.
• Generic: HuggingFace tokenizers can estimate counts for most open-weight models.

The visual layer that Graphify adds on top of these raw counts - turning numbers into maps of where tokens are concentrated and why - is what makes the difference between knowing a prompt is expensive and knowing which part to fix. Lesson 3 introduces that layer.