Model Routing

Learn how to route each AI request to the cheapest model that can handle it well. Intelligent model routing can reduce your API costs by 60-75% without sacrificing quality where it matters.

The Model Selection Problem

Using Claude Opus for every API request is like driving a Ferrari to the grocery store. It will get you there, but you are burning premium fuel for a trip that a compact car handles perfectly well. The same principle applies to AI models: the most powerful model is rarely the most cost-effective choice for every task.

Here is the reality most teams discover after their first month in production: 60-80% of all API requests are simple enough to be handled by the cheapest available model. Classification tasks, data extraction, yes/no questions, text formatting, and simple lookups do not need the reasoning power of a frontier model. They need speed, low cost, and consistent output.

The remaining 20-40% of requests are where you actually need the power of Sonnet or Opus. Complex multi-step reasoning, nuanced analysis, creative writing with specific constraints, and ambiguous tasks that require judgment - these are the requests that justify premium pricing. The key insight is that you should only pay premium prices for premium tasks.

Model routing is the practice of automatically directing each incoming request to the most appropriate (and usually cheapest) model that can deliver acceptable quality. Done well, it is the single highest-impact optimization you can make for token cost reduction.

The Model Tier Strategy

Before building a router, you need to define your model tiers. Each tier represents a level of capability and cost, with clear guidelines for what types of tasks belong to each:

The cost difference between tiers is dramatic. Haiku is roughly 19x cheaper than Opus on input tokens and 19x cheaper on output tokens. Even routing just half your traffic from Sonnet to Haiku saves 73% on that traffic. These savings compound quickly at scale.

Building a Router

The simplest and most effective starting point is a rule-based router. It examines the incoming request - task type, prompt length, category, keywords - and selects the appropriate model. No machine learning required.

1. Define Task Categories

   Map every API call in your system to a task type: classification, extraction, summarization, generation, analysis, or reasoning. Each category maps to a default tier.

2. Implement the Router Function

   Write a routing function that takes the task type and prompt characteristics, then returns the appropriate model identifier.

3. Add Complexity Scoring

   Estimate task difficulty from prompt characteristics: length, number of instructions, presence of ambiguous language, and required output format.

4. Test and Calibrate

   Run your router against historical requests and compare quality scores. Adjust tier boundaries until you find the right balance of cost and quality.

Here is a practical implementation of a rule-based router:

import anthropic

# Model tier definitions
MODELS = {
    "tier1": "claude-haiku-4-5-20250315",
    "tier2": "claude-sonnet-4-6-20250514",
    "tier3": "claude-opus-4-6-20250514",
}

# Task type to tier mapping
TASK_TIERS = {
    "classify":    "tier1",
    "extract":     "tier1",
    "format":      "tier1",
    "yes_no":      "tier1",
    "summarize":   "tier2",
    "code":        "tier2",
    "qa":          "tier2",
    "analyze":     "tier2",
    "reason":      "tier3",
    "research":    "tier3",
    "creative":    "tier3",
}

def route_request(task_type, prompt, **kwargs):
    """Route a request to the appropriate model tier."""

    # Step 1: Get the default tier for this task type
    tier = TASK_TIERS.get(task_type, "tier2")

    # Step 2: Adjust based on complexity signals
    complexity = estimate_complexity(prompt)

    if complexity > 0.8 and tier != "tier3":
        tier = "tier3"  # Upgrade complex requests
    elif complexity < 0.3 and tier == "tier2":
        tier = "tier1"  # Downgrade simple requests

    model = MODELS[tier]
    return model

def estimate_complexity(prompt):
    """Score prompt complexity from 0.0 to 1.0."""
    score = 0.0

    # Length-based signals
    word_count = len(prompt.split())
    if word_count > 500: score += 0.2
    if word_count > 1000: score += 0.2

    # Instruction complexity signals
    indicators = ["compare", "analyze", "evaluate",
                  "step by step", "pros and cons",
                  "trade-off", "nuance", "ambiguous"]
    matches = sum(1 for i in indicators if i in prompt.lower())
    score += min(matches * 0.15, 0.4)

    # Multi-part request signals
    numbered_steps = prompt.count("\n1.") + prompt.count("\n- ")
    if numbered_steps > 3: score += 0.2

    return min(score, 1.0)

This router is deliberately simple. It uses a dictionary lookup for the default tier, then adjusts based on a complexity score derived from the prompt itself. In production, this kind of straightforward approach handles the vast majority of routing decisions correctly.

The Cascade Pattern in Detail

The cascade pattern is a more sophisticated routing strategy: start with the cheapest model, check if the response meets a quality threshold, and escalate to a more expensive model only if needed. This guarantees you never overpay, because you only use expensive models when cheaper ones fail.

import anthropic

client = anthropic.Anthropic()

CASCADE_ORDER = [
    "claude-haiku-4-5-20250315",
    "claude-sonnet-4-6-20250514",
    "claude-opus-4-6-20250514",
]

def cascade_request(prompt, confidence_threshold=0.7):
    """Try models in order of cost, escalating if confidence is low."""

    for model in CASCADE_ORDER:
        response = client.messages.create(
            model=model,
            max_tokens=1024,
            messages=[{
                "role": "user",
                "content": prompt + "\n\nRate your confidence "
                          "in this answer from 0.0 to 1.0."
            }]
        )

        text = response.content[0].text
        confidence = extract_confidence(text)

        if confidence >= confidence_threshold:
            return {
                "response": text,
                "model_used": model,
                "confidence": confidence,
                "cascaded": model != CASCADE_ORDER[0],
            }

    # Final model response is always returned
    return {
        "response": text,
        "model_used": CASCADE_ORDER[-1],
        "confidence": confidence,
        "cascaded": True,
    }

def extract_confidence(text):
    """Parse confidence score from model response."""
    import re
    match = re.search(r"confidence[:\s]*([\d.]+)", text.lower())
    return float(match.group(1)) if match else 0.5

The cascade pattern works best when you have clear quality signals. Here is when to use cascading versus direct routing:

• Use cascading when quality is hard to predict upfront, when tasks vary widely in difficulty, or when you are willing to trade some latency for cost savings.
• Use direct routing when you already know the task complexity, when latency is critical, or when your rule-based router has been well-calibrated against production data.

Real-World Routing Distribution

When you analyze real production traffic, the distribution of task complexity is heavily skewed toward simple tasks. Here is what a typical routing distribution looks like after proper calibration:

When you calculate the weighted average, intelligent routing saves 60-75% compared to using Opus for everything. For a system spending $10,000/month on Opus, that is $6,000-7,500 in monthly savings - or $72,000-90,000 per year.

Monitoring Router Performance

A router is only as good as its monitoring. Without visibility into how your router performs, you cannot know if you are routing correctly or silently degrading quality. Track these four metrics continuously:

1. Model Distribution

   Track what percentage of requests go to each tier. If Opus is handling more than 10-15% of traffic, your router may be too conservative. If Haiku is handling more than 70%, verify that quality is not suffering.

2. Quality Scores Per Tier

   Measure response quality (user ratings, automated evals, or task-specific metrics) broken down by model tier. If Haiku's quality score drops below your threshold for certain task types, those tasks need to be upgraded to Sonnet.

3. Cost Per Request

   Track the average cost per request over time, broken down by task type and model. This is your primary optimization metric. It should decrease as you calibrate your router and increase your Haiku routing percentage.

4. Cascade Rate

   If using the cascade pattern, monitor how often requests escalate from Haiku to Sonnet or from Sonnet to Opus. A cascade rate above 30% means your initial routing is too aggressive - you are wasting money on failed attempts. Target a cascade rate under 15%.

# Simple router metrics tracker
class RouterMetrics:
    def __init__(self):
        self.requests = {"tier1": 0, "tier2": 0, "tier3": 0}
        self.cascades = 0
        self.total_cost = 0.0
        self.quality_scores = {"tier1": [], "tier2": [], "tier3": []}

    def log_request(self, tier, cost, quality, cascaded=False):
        self.requests[tier] += 1
        self.total_cost += cost
        self.quality_scores[tier].append(quality)
        if cascaded:
            self.cascades += 1

    def report(self):
        total = sum(self.requests.values())
        return {
            "distribution": {
                k: f"{v/total*100:.1f}%"
                for k, v in self.requests.items()
            },
            "avg_cost": self.total_cost / total,
            "cascade_rate": f"{self.cascades/total*100:.1f}%",
            "avg_quality": {
                k: f"{sum(v)/len(v):.2f}"
                for k, v in self.quality_scores.items()
                if v
            },
        }