Rate Limiting

Design rate limiting strategies that protect AI services from abuse, control costs, and ensure fair usage without impacting legitimate users.

Why Traditional Rate Limiting Falls Short

Standard requests-per-minute rate limiting does not work for AI APIs because not all requests are equal. A 10-token request costs 1000x less than a 100,000-token request. Effective AI rate limiting must account for computational cost.

Token-Based Rate Limiting

Rate limit by tokens consumed rather than requests made:

import redis, time

class TokenRateLimiter:
    def __init__(self, redis_client):
        self.redis = redis_client

    def check_and_consume(self, user_id, tokens_requested):
        key = f"rate:{user_id}:tokens"
        window = 60  # 1-minute window

        pipe = self.redis.pipeline()
        pipe.get(key)
        pipe.ttl(key)
        current, ttl = pipe.execute()

        current_usage = int(current or 0)
        limit = self.get_user_limit(user_id)

        if current_usage + tokens_requested > limit:
            return {
                "allowed": False,
                "remaining": max(0, limit - current_usage),
                "reset_in": ttl if ttl > 0 else window
            }

        # Consume tokens
        pipe = self.redis.pipeline()
        pipe.incrby(key, tokens_requested)
        if ttl < 0:
            pipe.expire(key, window)
        pipe.execute()

        return {"allowed": True,
                "remaining": limit - current_usage - tokens_requested}

Multi-Dimensional Rate Limiting

Apply rate limits across multiple dimensions simultaneously:

Cost Attack Prevention

Cost attacks deliberately send expensive requests to exhaust a victim's AI API budget. Defenses include:

• Max input length: Reject prompts exceeding a reasonable token count for your use case.
• Max output tokens: Set max_tokens per request to prevent runaway generation costs.
• Spending alerts: Notify users when they reach 50%, 80%, and 95% of their budget.
• Hard spending caps: Automatically disable API access when a spending threshold is reached.
• Anomaly detection: Flag sudden spikes in usage that may indicate a compromised key or cost attack.

Response Headers

Always return rate limit information in response headers so clients can self-regulate:

X-RateLimit-Limit-Requests: 60
X-RateLimit-Limit-Tokens: 100000
X-RateLimit-Remaining-Requests: 45
X-RateLimit-Remaining-Tokens: 72500
X-RateLimit-Reset: 1710500000
Retry-After: 30