Router & Gateway Pattern

The router pattern directs AI requests to the most appropriate model, handler, or processing pipeline based on characteristics of the input. It is how you build systems that use the right tool for each job - optimizing cost, latency, and quality simultaneously.

What Is Model Routing?

Not every request needs your most powerful (and most expensive) model. A simple greeting doesn't need Claude Opus. A complex legal analysis doesn't belong on a tiny model. Model routing is the pattern of analyzing an incoming request and directing it to the handler best suited to process it.

Think of it like a hospital triage system: a nurse evaluates each patient and routes them to the right department. Minor issues go to the general practitioner, urgent cases go to the emergency room, and specialist cases go to the appropriate specialist. The triage step is fast and cheap, but it dramatically improves the efficiency of the entire system.

In AI applications, routing decisions can be based on many factors: the complexity of the request, the intent of the user, the required accuracy, the cost budget, the expected latency, or even the language of the input. The key insight is that the routing decision itself should be fast and cheap - it should add minimal overhead while providing maximum value.

Classification-Based Routing

The most common routing approach uses a classifier to categorize incoming requests and route each category to a specialized handler. The classifier can be a traditional ML model (fast and cheap) or a small LLM (more flexible but slower).

How It Works

1. Incoming request arrives at the router
2. A lightweight classifier analyzes the request and assigns a category
3. The router maps the category to a specific handler (model, prompt template, or pipeline)
4. The handler processes the request and returns the result

Classification-based routing works best when you have well-defined categories with distinct characteristics. It struggles when categories overlap significantly or when the classification itself is ambiguous.

LLM-Based Routing

Instead of a traditional classifier, you can use a small, fast LLM to analyze the request and decide where to route it. This is more flexible than classification-based routing because the LLM can understand nuance and handle edge cases, but it adds latency and cost.

The routing LLM receives the user's request along with descriptions of available handlers and selects the best one. This is essentially a meta-prompt: "Given this request and these available specialists, which one should handle it?"

Embedding Similarity Routing

A fast, scalable approach that uses vector embeddings to match requests to handlers. You pre-compute embeddings for example requests in each category and route new requests to the category with the most similar examples. This avoids LLM calls entirely and can handle thousands of routes efficiently.

• Pre-compute embeddings for representative examples of each route
• Embed the incoming request using the same embedding model
• Find the nearest neighbors among the pre-computed embeddings
• Route to the handler associated with the closest match

Rule-Based Routing

The simplest and fastest routing approach: use deterministic rules based on request properties. No ML model needed, no API calls, no latency. Rule-based routing is ideal for clear-cut cases and serves as an excellent first layer before more sophisticated routing.

• Keyword matching: Route "refund", "billing", "payment" to the billing handler
• Language detection: Route non-English requests to a multilingual model
• Input length: Route very long inputs to models with larger context windows
• User tier: Route premium users to the best model, free users to the cheapest
• Content type: Route image inputs to vision models, audio to speech models

The API Gateway Pattern for AI

An AI API gateway sits between your application and multiple AI model providers, providing a unified interface. It handles routing, load balancing, rate limiting, fallbacks, logging, and cost tracking in one place. Think of it as an nginx or API gateway specifically designed for AI model APIs.

Key capabilities of an AI gateway:

• Unified API: One endpoint, one format, regardless of whether the request goes to Anthropic, OpenAI, Google, or a self-hosted model
• Smart routing: Route based on cost, latency, capability, or custom logic
• Rate limiting: Protect against runaway costs and API quota exhaustion
• Caching: Cache identical or semantically similar requests to save cost and latency
• Observability: Log every request, response, latency, cost, and error for debugging and analytics
• Fallback chains: Automatically retry with a different provider if the primary fails

Open-source AI gateways include LiteLLM (unified API across 100+ models), Portkey (routing, caching, and observability), and AI Gateway by Cloudflare (edge caching and rate limiting).

Code Example: Intent-Based Router

This router classifies user intent and routes to specialist handlers, each with its own optimized prompt and model configuration:

import anthropic
import json
from dataclasses import dataclass
from typing import Callable

client = anthropic.Anthropic()

@dataclass
class Route:
    """Defines a routing destination with its handler."""
    name: str
    description: str
    model: str
    system_prompt: str
    max_tokens: int

# Define specialist routes
ROUTES = {
    "code_help": Route(
        name="Code Help",
        description="Programming questions, debugging, code review",
        model="claude-sonnet-4-20250514",
        system_prompt="You are an expert programmer. Provide clear, "
                      "working code with explanations. Always include "
                      "error handling and best practices.",
        max_tokens=4000
    ),
    "creative_writing": Route(
        name="Creative Writing",
        description="Stories, poems, creative content generation",
        model="claude-sonnet-4-20250514",
        system_prompt="You are a creative writing assistant. Write "
                      "vivid, engaging content with strong voice and "
                      "narrative structure.",
        max_tokens=4000
    ),
    "data_analysis": Route(
        name="Data Analysis",
        description="Data questions, statistics, chart interpretation",
        model="claude-sonnet-4-20250514",
        system_prompt="You are a data analyst. Provide precise, "
                      "quantitative answers with methodology notes. "
                      "Use tables and structured formats.",
        max_tokens=3000
    ),
    "general": Route(
        name="General",
        description="Greetings, simple questions, chitchat",
        model="claude-haiku-4-20250514",
        system_prompt="You are a helpful, friendly assistant. "
                      "Keep responses concise and clear.",
        max_tokens=1000
    ),
}

def classify_intent(user_message: str) -> str:
    """Classify user intent using a fast, cheap model call."""
    route_descriptions = "\n".join(
        f"- {key}: {route.description}"
        for key, route in ROUTES.items()
    )

    response = client.messages.create(
        model="claude-haiku-4-20250514",  # Fast and cheap for classification
        max_tokens=50,
        messages=[{
            "role": "user",
            "content": (
                f"Classify this message into one category:\n\n"
                f"Categories:\n{route_descriptions}\n\n"
                f"Message: {user_message}\n\n"
                f"Return ONLY the category key (e.g., 'code_help'). Nothing else."
            )
        }]
    )
    intent = response.content[0].text.strip().lower()
    return intent if intent in ROUTES else "general"

def route_request(user_message: str) -> dict:
    """Route a request to the appropriate specialist handler."""
    # Step 1: Classify intent (fast, cheap)
    intent = classify_intent(user_message)
    route = ROUTES[intent]
    print(f"[Router] Classified as '{intent}' -> {route.name} ({route.model})")

    # Step 2: Process with specialist model and prompt
    response = client.messages.create(
        model=route.model,
        max_tokens=route.max_tokens,
        system=route.system_prompt,
        messages=[{"role": "user", "content": user_message}]
    )

    return {
        "intent": intent,
        "route": route.name,
        "model_used": route.model,
        "response": response.content[0].text,
        "tokens_used": response.usage.input_tokens + response.usage.output_tokens
    }

# Usage
result = route_request("Can you help me fix this Python TypeError?")
print(f"Route: {result['route']}")
print(f"Model: {result['model_used']}")
print(f"Response: {result['response'][:200]}...")

Code Example: Complexity-Based Router

This router estimates the complexity of a request and routes to increasingly capable (and expensive) models. Simple questions go to Haiku, medium complexity to Sonnet, and the hardest problems go to Opus:

import anthropic
import json

client = anthropic.Anthropic()

# Model tiers ordered by capability and cost
MODEL_TIERS = [
    {
        "name": "fast",
        "model": "claude-haiku-4-20250514",
        "max_complexity": 3,
        "cost_per_1k_input": 0.00025,
        "description": "Simple factual questions, greetings, short tasks"
    },
    {
        "name": "balanced",
        "model": "claude-sonnet-4-20250514",
        "max_complexity": 7,
        "cost_per_1k_input": 0.003,
        "description": "Analysis, coding, multi-step reasoning"
    },
    {
        "name": "powerful",
        "model": "claude-opus-4-20250514",
        "max_complexity": 10,
        "cost_per_1k_input": 0.015,
        "description": "Complex research, nuanced writing, hard problems"
    }
]

def estimate_complexity(message: str) -> dict:
    """Estimate request complexity on a 1-10 scale."""
    response = client.messages.create(
        model="claude-haiku-4-20250514",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": (
                "Rate the complexity of this request on a 1-10 scale.\n\n"
                "1-3: Simple (facts, greetings, yes/no, short answers)\n"
                "4-7: Medium (analysis, coding, explanations, comparisons)\n"
                "8-10: Complex (research, nuanced reasoning, creative work)\n\n"
                f"Request: {message}\n\n"
                'Return JSON: {"score": N, "reason": "brief explanation"}'
            )
        }]
    )
    return json.loads(response.content[0].text)

def complexity_router(message: str) -> dict:
    """Route based on estimated complexity."""
    # Step 1: Estimate complexity
    complexity = estimate_complexity(message)
    score = complexity["score"]

    # Step 2: Select model tier
    selected_tier = MODEL_TIERS[-1]  # Default to most powerful
    for tier in MODEL_TIERS:
        if score <= tier["max_complexity"]:
            selected_tier = tier
            break

    print(f"[Router] Complexity: {score}/10 ({complexity['reason']})")
    print(f"[Router] Selected: {selected_tier['name']} ({selected_tier['model']})")

    # Step 3: Process with selected model
    response = client.messages.create(
        model=selected_tier["model"],
        max_tokens=4000,
        messages=[{"role": "user", "content": message}]
    )

    input_tokens = response.usage.input_tokens
    estimated_cost = (input_tokens / 1000) * selected_tier["cost_per_1k_input"]

    return {
        "complexity_score": score,
        "complexity_reason": complexity["reason"],
        "tier": selected_tier["name"],
        "model": selected_tier["model"],
        "response": response.content[0].text,
        "estimated_cost": f"${estimated_cost:.6f}"
    }

# Examples showing different routing decisions
print(complexity_router("What is 2 + 2?"))
# -> fast tier (Haiku), complexity ~1

print(complexity_router("Write a Python function to merge two sorted lists"))
# -> balanced tier (Sonnet), complexity ~5

print(complexity_router(
    "Analyze the philosophical implications of Godel's incompleteness "
    "theorems on artificial general intelligence"
))
# -> powerful tier (Opus), complexity ~9

Load Balancing Across Model Providers

In production, you often use multiple AI providers for redundancy and cost optimization. Load balancing distributes requests across providers while respecting rate limits, cost budgets, and quality requirements.

• Round-robin: Distribute requests evenly across providers. Simple but doesn't account for provider differences.
• Weighted distribution: Send more traffic to preferred providers. Useful when one provider offers better pricing or quality.
• Least-connections: Route to the provider with the fewest in-flight requests. Helps maintain consistent latency.
• Cost-optimized: Track spending per provider and route to stay within budget allocations.
• Latency-based: Route to the provider currently responding fastest. Requires continuous latency monitoring.

A/B Testing with Routing

Routing infrastructure makes A/B testing AI models straightforward. Split traffic between model versions, collect quality metrics, and make data-driven decisions about which model to promote.

Key considerations for AI A/B testing:

• Consistent user assignment: Hash the user ID to ensure the same user always gets the same model variant during a test
• Quality metrics: Define measurable quality criteria before starting the test (user satisfaction, task completion rate, response accuracy)
• Cost tracking: Compare not just quality but also cost per request for each variant
• Statistical significance: Run tests long enough to get meaningful results, especially for tasks with high variance
• Gradual rollout: Start with 5-10% of traffic on the new variant and increase gradually as confidence grows

Fallback Routing

When a provider goes down or returns errors, fallback routing automatically redirects traffic to an alternative provider. This is critical for production systems where downtime is unacceptable.

A robust fallback strategy includes:

• A prioritized list of providers for each route
• Health checks to detect provider issues before they affect users
• Circuit breakers to stop sending traffic to a failing provider
• Alerting when fallbacks are activated so you can investigate

Routing Strategies Comparison