Scaling Azure OpenAI

Master rate limits, quota management, load balancing, and retry strategies for handling production-scale traffic on Azure OpenAI Service.

Understanding Quotas and Limits

Load Balancing with API Management

<!-- APIM policy for round-robin load balancing -->
<policies>
  <inbound>
    <set-variable name="backendIndex"
      value="@(new Random().Next(0, 3))" />
    <choose>
      <when condition="@(context.Variables
        .GetValueOrDefault<int>("backendIndex") == 0)">
        <set-backend-service
          base-url="https://oai-eastus.openai.azure.com/" />
      </when>
      <when condition="@(context.Variables
        .GetValueOrDefault<int>("backendIndex") == 1)">
        <set-backend-service
          base-url="https://oai-westus.openai.azure.com/" />
      </when>
      <otherwise>
        <set-backend-service
          base-url="https://oai-northeu.openai.azure.com/" />
      </otherwise>
    </choose>
  </inbound>
</policies>

Retry and Backoff Strategies

• 429 responses: Respect the Retry-After header returned with rate limit errors
• Exponential backoff: Implement exponential backoff with jitter for transient errors
• Circuit breaker: Open circuit after repeated failures to prevent cascading issues
• Fallback regions: Route to secondary regions when primary region is rate-limited
• Request queuing: Queue requests during traffic spikes and process at sustainable rate

Scaling Strategies