The Complete Guide to Handling Rate Limits in AI APIs (2026)

Introduction

Rate limits are the single most common error class in AI applications. Not context window exceeded. Not malformed JSON. Not deprecated models. Rate limits — the 429s that fire during peak traffic, the token-per-minute walls you hit on your first large context request, the daily quotas that expire your batch jobs at midnight UTC. Every team building on AI APIs eventually spends a full afternoon rate-limited, watching their queue back up while a Slack thread debates whether to upgrade tiers or refactor the workload.

This guide is the cross-cutting reference for handling that class of problem. It covers what rate limits are, how each major provider structures them differently, how to detect them cleanly, how to back off correctly, when to add circuit breakers, when to horizontally scale, and when to reach for provisioned throughput. It cross-references every rate-limit error page in our errors library across all 19 verticals we've documented.

If you're triaging an active rate limit incident, skip to Chapter 4 (Detection) and Chapter 5 (Backoff). If you're architecting a new AI workload, read Chapter 2 (Taxonomy) and Chapter 12 (The mature strategy) to understand the design space. If you're evaluating whether to move to provisioned throughput, jump to Chapter 8. And if you're building an agent, Chapter 9 is where the specific failure modes for tool-loop workloads live.

Why AI rate limits are worse than traditional API rate limits

Rate limits aren't new. Every API has them. But AI providers are unique in three ways that make rate limit handling harder than a traditional REST API:

1. Multiple concurrent meters. A single request can hit RPM, TPM, ITPM, OTPM, or TPD depending on its size and your provider. Traditional APIs have RPM. AI APIs typically have four or more overlapping meters.
2. Token accounting is non-obvious. The token count you're billed against isn't the character count of your prompt — it depends on the tokenizer, images and audio are converted to token equivalents, and reasoning tokens don't appear in your response.
3. Rate limits interact with cost. On a REST API, being rate limited costs you time. On an AI API, hitting a rate limit often forces a fallback that costs 3-10x more per request. Rate limit handling is cost handling.

This third point is the key insight most teams miss. Rate limits and cost are not separate problems — they're the same problem seen from different angles. A workload that never hits its rate limits typically has room to reduce cost (you're overprovisioned). A workload that constantly hits rate limits is quietly overspending on fallbacks. The mature strategy is to sit in a narrow operating band that respects both meters — and this guide is how you find it.

The rate limit taxonomy

Before you can handle rate limits, you have to know which one is firing. The nomenclature is fragmented across providers, so here's the unified reference.

The meter your workload hits depends on its shape

Two workloads with the same total token consumption can hit completely different rate limit meters. Consider two systems both processing 10 million tokens per hour:

• Workload A: 10,000 requests per hour, average 1,000 tokens per request. Hits RPM first.
• Workload B: 100 requests per hour, average 100,000 tokens per request. Hits TPM (or ITPM) first, never touches RPM.

Same total consumption, completely different rate-limit exposure. Understanding which meter you're going to hit is the first step to designing around it. Run your traffic through your provider's dashboard for a week. Look at requests-per-minute AND tokens-per-minute AND input-tokens-per-minute if your provider splits them. The one you hit first is the one you optimize first.

Free tier vs paid tier vs custom tier

Every major provider structures tiers similarly:

• Free tier: Low rate limits, sometimes daily caps. Not for production.
• Automatic paid tiers: Rate limits scale automatically based on your usage history and payment activity. Anthropic's tier 1-4 system, OpenAI's automatic tier system, Google's paid tier all work this way.
• Custom / negotiated: Enterprise contracts with custom rate limits. Available on all major providers via account team contact.
• Provisioned throughput: Fixed capacity reservation (Bedrock PT, Azure PTUs). See Chapter 8.

Most production teams live in the automatic paid tier plateau, with a few enterprises on custom or PT. The tier structure is why your limits can suddenly increase over time — you're being auto-promoted based on payment history — and why a new project on a new API key can hit rate limits at scale even though your main workload doesn't.

How to identify which meter fired when you get a 429

The 429 response body usually tells you. Anthropic returns error.type: "rate_limit_error" with a message specifying the meter. OpenAI returns messages like "Rate limit reached for requests" vs "Rate limit reached for tokens". Gemini returns "Quota exceeded for aiplatform.googleapis.com/generate_content_requests".

Log the full error body on every 429. Don't just log "got 429" — the specific meter matters for choosing your response. A workload hitting TPM needs to reduce tokens (prompt caching, shorter context). A workload hitting RPM needs to reduce request rate (client-side rate limiter, request batching). A workload hitting ITPM specifically needs input-side optimization (RAG chunking, prefix caching). Wrong diagnosis → wrong fix → the problem persists.

The two-clock problem

Every rate limit is defined against a specific time window. RPM = requests in the last 60 seconds. TPM = tokens in the last 60 seconds. TPD = tokens in the last 24 hours. Providers use sliding windows (most common) or fixed windows (rare, but check your provider's docs).

Sliding windows mean your rate limit resets continuously — the 60-second-ago request expires from the counter this second. Fixed windows mean your rate limit resets at wall-clock boundaries (top of every minute). The behavior at burst boundaries differs meaningfully. If your provider uses fixed windows and you send 100 requests at 12:00:59 and another 100 at 12:01:01, you might hit rate limits despite averaging 100 RPM overall.

Provider-by-provider rate limit shape

Each provider structures rate limits differently. This chapter is the field guide.

Anthropic Claude — ITPM/OTPM split with tiers

Anthropic pioneered the input/output token split. As of 2026, Claude enforces four meters simultaneously per API key: RPM, ITPM (input tokens/min), OTPM (output tokens/min), TPD (tokens/day on lower tiers). Tier 1 → Tier 4 automatic progression based on payment activity. Batches API has its own quota separate from standard rate limits.

Common Claude rate-limit surprises:

• ITPM exceeded on large-context RAG workloads — you're not hitting RPM or OTPM, just ITPM
• RPM exceeded during agent workloads where a single user turn generates many tool calls
• TPD exceeded on Tier 1 for high-volume overnight batch workloads

Full reference: our Claude API pillar guide.

OpenAI — TPM primary, RPM secondary

OpenAI's rate limit structure is simpler: TPM (input + output combined) and RPM per model. Each model has its own limits (GPT-5.6 Terra has separate limits from Sol, from Luna). Batch API has separate quotas. The Responses API uses the same underlying limits as chat completions.

Common OpenAI rate-limit patterns:

• TPM exceeded on medium-context high-throughput workloads
• Generic 429 when RPM is hit — often during burst traffic
• Batch API expired when jobs don't complete in 24 hours

Full reference: our OpenAI API pillar guide.

Google Gemini — quota system

Gemini calls its rate limits quotas. Separate quota for generate_content requests, embed_content requests, and file uploads. Each quota is per-project on Google Cloud. Free tier (AI Studio) has aggressive limits; paid tier (Vertex AI) has higher limits with a straightforward request path for upgrades.

• generate_content quota exceeded
• Batch prediction quota exceeded
• Context cache TTL expiration — cache expiring doesn't rate limit but forces recreation, which does count against quota

Full reference: our Gemini API pillar guide.

Amazon Bedrock — model-specific quotas

Bedrock quotas are per-model, per-region. Requesting a quota increase requires an AWS Service Quotas request. The 2026 landscape added cross-region inference profiles that give higher throughput for supported models by routing across regions transparently.

• Model invocation quota exceeded
• PT commitment mismatch

Full reference: our Amazon Bedrock pillar guide.

Azure OpenAI — PTUs and standard tiers

Azure OpenAI has two paths: Standard (pay-as-you-go) with TPM/RPM limits per deployment, and Provisioned (PTUs) for guaranteed throughput. PTU minimums are model-specific — GPT-5.6 Terra requires 100 PTUs minimum for Global deployment.

• TPM quota exceeded
• PTU minimum commitment not met

Full reference: our Azure OpenAI pillar guide.

The rate-limit shape summary

For picking a provider based on rate limits: Anthropic Claude has the most granular metering (four meters), which is more information but more complexity. OpenAI has the simplest metering (TPM/RPM per model). Gemini has quota-based with straightforward increases. Bedrock is per-region with cross-region profiles for higher throughput. Azure OpenAI is the go-to for enterprises wanting PTU-based guaranteed capacity.

Detecting rate limits — headers & codes

Every provider tells you when you're rate limited. Not every provider tells you the same way. This chapter is the reference for what you get back and how to parse it correctly.

Status codes

All major providers return 429 Too Many Requests for rate limit violations. The 429 body typically contains an error code and a human-readable message. But the machine-readable metadata you actually want is in the response headers.

Response headers you should always parse

Every request from a rate-limit-conscious client should log these headers. Not just failed requests — succeeded requests too. The value of x-ratelimit-remaining-tokens on a successful request tells you how close you are to the ceiling. If it's dropping fast, back off proactively before the 429 fires.

The proactive vs reactive distinction

Two rate-limit-handling philosophies:

• Reactive — send requests until you get 429, then back off and retry. Simple. Works fine for low-volume workloads. Wastes provider capacity when many clients hit the wall simultaneously (thundering herd).
• Proactive — track your remaining budget from response headers. Slow down before you hit the wall. Never triggers 429s in the happy path.

Proactive is strictly better once your traffic is high enough that reactive causes noticeable degradation. The implementation is a client-side rate limiter (token bucket) with a bucket size derived from the provider's response headers. Refill rate = provider's per-minute limit / 60 (with safety margin, typically 80-90% of published limits).

Parsing retry-after correctly

The retry-after header can be either a delay in seconds (integer) or an HTTP-date. Most providers use integer seconds, but a robust parser handles both:

from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def parse_retry_after(header_value: str) -> float:
    """Return seconds to wait before retry."""
    if not header_value:
        return 1.0  # default fallback

    try:
        # Integer seconds
        return float(header_value)
    except ValueError:
        pass

    try:
        # HTTP-date
        target = parsedate_to_datetime(header_value)
        now = datetime.now(timezone.utc)
        return max(0.0, (target - now).total_seconds())
    except (TypeError, ValueError):
        return 60.0  # bad header, be conservative

When retry-after is absent from a 429 response (rare but happens), fall back to exponential backoff — see Chapter 5.

Exponential backoff done right

Exponential backoff is table stakes for API clients. Every SDK ships some version of it. Most implementations are naive in ways that make thundering herd worse, not better. This chapter is the reference for what a production-grade backoff actually looks like.

The naive version and why it fails

# Naive backoff — DON'T do this
def call_with_retry(fn, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return fn()
        except RateLimitError:
            wait = 2 ** attempt  # 1, 2, 4, 8, 16 seconds
            time.sleep(wait)
    raise RuntimeError("Max attempts exceeded")

Two problems:

1. Thundering herd. All clients that hit 429 at the same time back off for exactly 1 second, then 2 seconds, then 4 seconds. They all retry simultaneously, hitting the provider with a wave of requests each time.
2. Ignoring retry-after. The provider told you when to retry. The naive version ignores that hint.

The right version: full jitter + retry-after respect

import random
import time
from typing import Callable, TypeVar

T = TypeVar("T")

def call_with_backoff(
    fn: Callable[[], T],
    max_attempts: int = 6,
    base: float = 1.0,
    cap: float = 60.0,
) -> T:
    """Exponential backoff with full jitter, respecting retry-after."""
    for attempt in range(max_attempts):
        try:
            return fn()
        except RateLimitError as e:
            if attempt == max_attempts - 1:
                raise

            # Prefer server-provided retry-after
            retry_after = getattr(e, "retry_after", None)
            if retry_after and retry_after > 0:
                wait = min(retry_after, cap)
            else:
                # Full jitter — random between 0 and exponentially growing cap
                wait = random.uniform(0, min(cap, base * (2 ** attempt)))

            time.sleep(wait)
    raise RuntimeError("Unreachable")

Key properties:

• Full jitter — random.uniform(0, backoff_cap) spreads retries across the window, eliminating thundering herd
• Respects retry-after — when the server tells you when to retry, listen
• Bounded backoff — cap prevents extremely long waits (e.g., 8 minutes on attempt 8)
• Retry budget — max_attempts prevents infinite loops on genuinely down services

What backoff parameters to actually use

Starting defaults that work for most workloads:

• Base delay: 1.0 seconds
• Cap: 60.0 seconds (some workloads go higher; 60s is a good baseline)
• Max attempts: 5-8 for user-facing requests, up to 20 for background batch jobs

Adjust based on your workload's latency budget. A user-facing chatbot can't wait 60 seconds — reduce cap to 15 seconds and max_attempts to 3. An overnight batch job can be more patient — increase both.

The retry budget concept

Beyond per-request retry limits, a mature system also enforces a retry budget at the application level. If more than N% of requests are being retried in a given window, further retries are suppressed. This prevents cascading failure — if the underlying provider is degraded, you don't spend the next hour piling retries on top of retries.

Google's SRE workbook covers the pattern in detail. The simple version: track (retries / total_requests) in a rolling window. Once it exceeds a threshold (e.g., 10%), fail fast on further errors for a short cool-down period. This is close to a circuit breaker — see Chapter 6.

Circuit breakers — when retries make it worse

Backoff plus retry handles transient rate limits. It does not handle sustained provider degradation. When a provider is genuinely down or severely rate-limiting, continuing to retry just piles work on an already-broken service. This is where circuit breakers earn their keep.

The three states of a circuit breaker

• Closed — requests flow through normally. Failures are counted. If failures exceed threshold in window, transition to Open.
• Open — requests fail immediately without hitting the provider. After a cool-down period, transition to Half-open.
• Half-open — allow a small number of test requests through. If they succeed, transition back to Closed. If they fail, transition back to Open.

Minimal implementation

from dataclasses import dataclass, field
from time import monotonic
from enum import Enum

class State(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@dataclass
class CircuitBreaker:
    failure_threshold: int = 5      # failures in window to trip
    cool_down_seconds: float = 60.0  # time to wait before half-open
    window_seconds: float = 60.0     # window to count failures

    state: State = State.CLOSED
    failures: list = field(default_factory=list)
    opened_at: float = 0.0

    def call(self, fn):
        now = monotonic()

        # Purge old failures
        self.failures = [t for t in self.failures if now - t < self.window_seconds]

        if self.state == State.OPEN:
            if now - self.opened_at >= self.cool_down_seconds:
                self.state = State.HALF_OPEN
            else:
                raise CircuitBreakerOpen()

        try:
            result = fn()
            if self.state == State.HALF_OPEN:
                self.state = State.CLOSED
                self.failures.clear()
            return result
        except Exception:
            self.failures.append(now)
            if len(self.failures) >= self.failure_threshold:
                self.state = State.OPEN
                self.opened_at = now
            raise

What to trip on

Not every error should trip the breaker. Categorize:

• Trip on: 5xx errors, sustained 429s (5+ in a minute), timeouts
• Don't trip on: 4xx errors that indicate client bugs (400 bad request, 401 unauthorized) — these won't succeed on retry regardless
• Debatable: single 429s — these might be normal traffic bursts, not sustained problems

A mature client separates these categories. A 5xx or timeout trips the breaker aggressively. A 4xx surfaces to the caller immediately without touching the breaker. A single 429 backs off but doesn't count toward breaker failures until you've seen several in a row.

Per-endpoint vs global circuit breakers

If you use multiple providers, run separate circuit breakers per provider (or per model). When Anthropic is degraded, you want to stop hitting Anthropic — but that doesn't mean OpenAI is degraded. A per-endpoint breaker lets one provider fail while others continue serving requests.

This pairs naturally with cross-provider fallback (Chapter 7). When the Claude breaker trips open, requests transparently fail over to a GPT-5.6 fallback. Once the Claude breaker returns to closed after the cool-down, traffic returns to Claude. This is the pattern that lets applications tolerate multi-hour provider outages without user-visible impact.

Horizontal scaling — keys, regions, providers

Vertical scaling (higher tier, PTU) has limits. Eventually you're at the top tier and still hitting walls. Horizontal scaling — spreading traffic across API keys, regions, and providers — is where teams get past those walls.

Multiple API keys within a provider

Every major provider limits by API key (with some also having org-level ceilings above that). Splitting traffic across N keys often gives you N x per-key capacity, subject to org-level caps. Read your provider's TOS before scaling this pattern — some providers explicitly prohibit multi-key sharding on a single account.

Where it's allowed and useful:

• Different keys for different environments (production / staging / dev) with independent rate limits
• Different keys for different teams / features to isolate their rate-limit exposure
• Keys on different orgs when the org-level cap is the binding constraint

Cross-region routing

Some providers rate-limit per region. Others don't. The 2026 landscape:

• Anthropic: Rate limits are per-org globally, not per-region. Region choice doesn't help capacity.
• OpenAI: Similar — org-level rate limits regardless of region.
• AWS Bedrock: Per-region quotas. Cross-region inference profiles aggregate capacity across US, EU, or APAC regions transparently.
• Azure OpenAI: Per-deployment quotas. Multi-region deployments give you N x capacity.
• Google Gemini via Vertex AI: Per-project quotas. Multi-project setup can help.

For Bedrock and Azure, cross-region routing is one of the most impactful capacity multipliers available. Cross-region inference profiles on Bedrock are especially attractive because they don't require any application-side work — you just switch to the profile ID and Bedrock does the routing for you.

Cross-provider fallback

The most powerful horizontal scaling pattern: route to a different provider entirely when your primary is rate-limited. Requires:

• Multiple provider integrations (or a gateway that abstracts them)
• Cross-provider prompt compatibility (or careful prompt tuning per provider)
• Instrumentation to track which provider served each request
• Awareness that quality varies subtly per provider — an Opus response reads differently than a GPT-5.6 response

Tools that make cross-provider fallback easy: our gateway comparison covers LiteLLM, OpenRouter, and Portkey. Each handles fallback slightly differently. LiteLLM's config supports multiple fallback strategies (generic, content-policy-specific, context-window-specific, budget-specific). See the LiteLLM fallback cost spiral page for the specific configuration to avoid.

The trade-off nobody mentions

Horizontal scaling adds complexity. Multiple keys mean multiple secrets to rotate. Multi-region means multi-region deployment. Cross-provider means testing and monitoring across providers. Every additional dimension of scale is another dimension of operational surface area.

The right time to horizontal-scale is when a single dimension is genuinely maxed out. Don't preemptively add complexity — take the current provider to its highest sustainable tier first, use prompt caching (Chapter 3 of the Cost Playbook) to reduce token pressure, and only then look at horizontal scaling.

The "burst absorber" pattern

Bursty workloads (traffic spikes 3-10x baseline for short windows) are the hardest to horizontally scale. A single-key setup at 80% baseline utilization hits 100%+ on bursts and starts rate-limiting. The classic burst-absorber pattern:

1. Primary key at 60% baseline utilization — leaves headroom for bursts
2. Secondary key at 0% baseline — only activated during burst detection
3. Burst detection at client side — when incoming request rate exceeds threshold, route new requests to secondary
4. Return to primary — after burst subsides, drain new traffic back to primary key

This gives you 2x effective capacity during bursts without paying for 2x baseline. LiteLLM and Portkey both support this pattern via load balancing configs; direct provider SDK usage requires wrapper code.

The "hot key" anti-pattern

A common bug we see: teams add multiple API keys and load-balance randomly, but one key inherits all the "sticky sessions" (streaming responses, long-running batch jobs) while the others sit idle. Random load balancing across keys doesn't help if the workload is stateful.

Fix: hash sessions to keys deterministically, OR use round-robin at the request level rather than session level. For streaming workloads, accept that per-key utilization won't be perfectly even and provision for the peak, not the average.

Provisioned throughput vs on-demand

Provisioned throughput — Bedrock PT, Azure PTUs — trades upfront commitment for guaranteed capacity. Cost math is favorable at high sustained utilization; unfavorable at bursty or low utilization. Getting this decision right is a 30-50% cost lever for the right workload.

We covered this in depth in the Cost Optimization Playbook Chapter 8. The rate-limit-specific angle:

PT / PTU eliminates specific rate limit classes

On-demand pricing lives with variable rate limits — capacity depends on shared pool utilization at the provider. During peak hours, a busy provider can degrade for everyone on shared capacity.

PT / PTU reserves dedicated capacity. Your rate limit becomes purely a function of what you reserved. If you paid for 100K tokens/second, you get 100K tokens/second regardless of what other customers are doing. This is the "guaranteed throughput" pitch, and it's real.

When PT / PTU makes sense for rate limit reasons

• Latency SLAs that on-demand rate limits can't guarantee
• Peak load prediction for scheduled workloads (e.g., overnight batch processing that must complete by 6am)
• Regulatory requirements for capacity commitments
• Consistent workloads at scale — Series B+ companies with predictable AI usage patterns

When it doesn't

• Bursty startup traffic — utilization rarely hits the 60-70% break-even
• Diurnal workloads — 12 hours of activity means 50% average utilization
• Experimental workloads — you don't yet know if the feature will drive usage
• Small teams — administrative overhead of PT management is real

Bedrock cross-region inference profiles as a middle ground

Cross-region inference profiles give higher on-demand throughput without PT commitment. If you're on Bedrock and hitting rate limits, this is the first thing to try before moving to PT. It's a config change, not an infrastructure change, and it aggregates capacity across US/EU/APAC regions transparently.

Azure PTU minimums

Azure PTUs have per-model minimum commitments. GPT-5.6 Terra requires 100 PTUs minimum for Global deployment. Below the minimum, PTU isn't available. This eliminates PTU as an option for small teams — the entry point is meaningful spend, not a small commitment.

The decision framework

Simplest test: run your on-demand workload for a month. Track sustained utilization vs your provider's rate limits. If you're at 40%+ of your top-tier limits consistently, and your workload will be around for another year, PT starts penciling. Below 30%, on-demand plus horizontal scaling is almost always cheaper.

Rate limits in agent workloads

Agent applications hit rate limits harder than any other workload class. A single user turn generates multiple model calls — one to plan, one to invoke each tool, one to synthesize the result. Five tools in a turn = five requests. Ten users doing this simultaneously = 50 requests in a burst.

Why agent workloads pressure RPM more than TPM

Agent requests tend to be small individually (10-20K token context) but frequent (5-20 per user turn). This burns RPM hard while barely touching TPM. Teams that architected for TPM headroom get surprised when RPM is the binding constraint.

First rule of agent rate-limit management: cap iterations per turn. Every agent framework supports this — LangChain's recursion_limit, LangGraph's recursion_limit, LlamaIndex Workflows' step count. Setting the cap explicitly prevents runaway loops from burning RPM.

Subagent context isolation reduces rate limit pressure

See our coding tools comparison — context management section. Subagents (Claude Code, LangGraph subgraphs, LlamaIndex nested workflows) let you delegate heavy work to isolated contexts. The main agent doesn't consume RPM for the entire tool sequence — the subagent does, and the main agent only sees the summary.

This isn't primarily a cost optimization — it's a rate limit management technique. Subagents let you parallelize tool sequences that would otherwise serialize into a single RPM-blowing chain.

Batching tool calls

Many agent frameworks let you invoke multiple tools in parallel within a single model turn. If your problem can be decomposed into independent subtasks, this cuts request count dramatically. Ten sequential tool calls becomes one turn with ten parallel invocations — that's 10x reduction in RPM for the same work.

Not all workloads decompose this way. Chain-of-thought problems where each tool depends on the previous can't be parallelized. But surprisingly many can, and most frameworks make this a config choice rather than a rewrite.

Agent-specific fallback strategies

When an agent hits a rate limit mid-turn, the failure mode is worse than a single-request failure. You've already invoked half the tools; a rate limit on the synthesis call means you paid for the tool work and got no output.

Best practice: cache tool outputs by input hash. If the agent has to retry after a rate limit, it can skip the tools it already ran successfully. LangChain, LangGraph, and LlamaIndex all support this pattern via memoization or explicit caching layers. See LangGraph checkpointer for the specific implementation.

Cursor and Claude Code as agent workload examples

Cursor's $20 Pro credit cap and Claude Code's auto-compact thrashing are both agent-workload rate limit patterns manifesting at the IDE tool layer. The underlying pattern is the same: many small requests per user action compound faster than teams expect. See the next chapter for the tool-specific patterns.

Worked example: a customer support agent

Concrete numbers to make this real. Consider a customer support agent that handles user queries by:

1. Classifying intent (1 model call)
2. Retrieving relevant knowledge base articles (1 tool call → 1 model call to synthesize)
3. Checking order status if applicable (1 tool call → 1 model call)
4. Generating final response (1 model call)

That's 5 model calls per user turn. Ten concurrent users doing this = 50 requests in a burst. On Claude Tier 2 (1,000 RPM), you have 20 seconds of headroom before you hit RPM. If turns take 15 seconds end-to-end, you're at 133% of capacity during peak minutes.

Three fixes, ordered by cost/benefit:

1. Parallelize independent tool calls — steps 2 and 3 can run concurrently in the same model turn. Cuts request count from 5 to 3. New capacity headroom: 33 seconds.
2. Cache the classifier — intent classification is often deterministic for common queries. Cache-hit skips step 1 entirely on ~40% of queries.
3. Route classifier to Haiku or a smaller model — separate rate limit pool, plus 5-10x cheaper per call.

Result: peak RPM drops from 50 to ~18, comfortably within Tier 2. Same user experience, no tier upgrade required. This kind of architectural tuning consistently outperforms tier upgrades for agent workloads.

Rate limits in IDE tools

IDE tools (Cursor, Claude Code, GitHub Copilot) add their own rate limit layer on top of the underlying provider limits. When you hit one, understanding which layer fired is critical.

Cursor — credit-based metering

Cursor's Pro plan gives you $20/mo of frontier-model credits. Composer 2.5 requests are effectively free within plan; Opus 4.7 burns credits ~5x faster than Sonnet 4.7. See the usage cap exceeded page for the specific failure mode.

Cursor also has a lesser-known issue where Auto mode's fallback logic can shuffle you between rate-limited models. If Auto mode is failing, switch to a specific model in the picker rather than trusting Auto's selection.

Claude Code — usage tiers within Anthropic subscription

Claude Code inherits Anthropic subscription limits (Claude Pro, Max 5x, Max 20x). Hitting the Claude Pro daily limit triggers the underlying Claude API rate limits — see the ITPM/OTPM/RPM error pages we already linked in Chapter 3.

For interactive Claude Code use, the specific pattern to watch is auto-compact thrashing — repeated context compaction is not a rate limit per se, but it degrades interaction quality in a way that feels like a rate limit and is worth diagnosing.

GitHub Copilot — session rate limits + AI Credits

Copilot's rate limit landscape changed materially in June 2026 when it moved to usage-based AI Credits. Two distinct rate limit surfaces now:

• Agent Mode session rate limit — time-based, tied to your account, not per-chat. Opening a new chat does NOT reset it.
• Global rate limit with premium quota remaining — VS Code bug #311788, the error text and quota display disagree.

Copilot Agent Mode uses 5-20x more tokens per request than Chat mode, so the session rate limit is specifically designed to protect infrastructure — not just to encourage upgrades.

The tool-layer rate limits don't obey the underlying provider's limits

This is the key insight. Cursor's $20 credit cap has nothing to do with Anthropic's ITPM. Copilot's session rate limit has nothing to do with OpenAI's TPM. IDE tools implement their own accounting on top of the underlying provider, and their rate limits fire for their own reasons.

Practical implication: if your Cursor / Claude Code / Copilot workflow is rate-limited, upgrading your direct Anthropic or OpenAI API key does not help. You need to upgrade the tool's subscription tier (or switch tools).

Rate limits at the gateway layer

Gateways (LiteLLM, OpenRouter, Portkey) add another rate limit layer. Their rate limits are independent from the underlying provider's — sometimes higher, sometimes lower, sometimes structured completely differently.

OpenRouter — free tier limits are aggressive

OpenRouter's free tier is 20 req/min hard cap and 50-1000 req/day. Purchased credits raise the daily floor to 1000 but don't change the per-minute cap. See the OpenRouter rate limits page for the specific breakdown.

This makes OpenRouter's free tier unsuitable for production. For production, use OpenRouter's paid tiers or BYOK (Bring Your Own Key) — BYOK routes through your provider account, subject to your provider's rate limits rather than OpenRouter's shared pool.

LiteLLM — virtual keys with per-key rate limits

LiteLLM's virtual keys can carry per-key rate limits (RPM/TPM) and budget caps. See virtual key budget exceeded page. This lets you isolate rate limit exposure per team or per feature — the growth team's runaway feature can't drain the whole org's budget.

LiteLLM v1.85+ added budget fallbacks: when a virtual key hits its per-model budget, the request reroutes to a cheaper model instead of failing. Combined with rate limit fallbacks, this creates a graceful degradation path.

Portkey — virtual keys with independent rate limits

Portkey's virtual keys also carry independent rate limits and budgets. Similar model to LiteLLM but with a hosted UI for configuration.

Portkey adds workspace-level rate limits above individual key limits — a global safety net. Configure workspace limits above the sum of individual key limits so they only fire on org-wide runaway.

Gateway fallback strategies for rate limits

The pattern we recommended in Chapter 7 — cross-provider fallback on rate limit — is where gateways earn their keep. All three (LiteLLM, OpenRouter, Portkey) support this to varying degrees. Our gateway comparison post covers the specific configuration for each.

Key caveat: fallback across providers changes output quality. A Sonnet-tuned prompt won't produce identical outputs on GPT-5.6. Design your workload to tolerate this variation, or restrict fallback to same-family models (Sonnet → Opus is safer than Sonnet → GPT-5.6).

The rate-limit tax gateways add

Gateways add latency to every request (~10-40ms depending on which). At high request volumes, this is meaningful — an extra 20ms on a 500ms baseline is 4% throughput reduction. For rate-limit-bound workloads, that 4% can be the difference between fitting inside a tier and getting rate limited.

The trade-off usually still favors the gateway. Cross-provider fallback and observability are worth more than 4% throughput. But instrument the added latency on your own workload — some gateway configurations add 100ms+ on cold-start requests, which is meaningful for real-time UX.

Gateway-level virtual key rate limits vs provider limits

Virtual keys with RPM/TPM caps below the provider's actual limit give you two layers of protection:

1. Per-team virtual keys prevent one team from consuming the whole org's provider quota
2. Global provider quota still bounds the org as a whole

Set virtual key caps to (provider limit / number of teams) * 1.2 for a 20% safety margin. This gives predictable per-team throughput while allowing occasional bursts.

The mature rate-limit strategy

Everything above is components. This chapter is how they fit together. What does a well-architected AI application actually do when it comes to rate limits?

The mature rate-limit strategy has six layers, applied in order:

Layer 1 — Instrument before you optimize

Log x-ratelimit-* and anthropic-ratelimit-* headers on every request. Track requests per minute, tokens per minute, input tokens per minute separately per workload. Alert on approach to any meter's threshold (80% is a good default). See our Cost Playbook Chapter 11 for the specific instrumentation pattern.

Layer 2 — Client-side rate limiting at 80-90% of provider limit

Never send at 100% of your provider's rate limit. Bursts happen; the buffer between your soft limit and the provider's hard limit absorbs them. Client-side token buckets (see ratelimit Python package, bottleneck in JavaScript) are the standard tool.

Layer 3 — Exponential backoff with full jitter on 429

Chapter 5's implementation. Respect retry-after. Cap wait times. Bound retry count. Log every retry with its wait time and outcome.

Layer 4 — Circuit breakers for sustained degradation

Chapter 6's pattern. Per-provider, per-model breakers with configurable failure thresholds. Trip on 5xx and sustained 429s. Half-open recovery.

Layer 5 — Horizontal scaling when a single provider is exhausted

Chapter 7's patterns. Multiple keys within tier, multi-region for Bedrock and Azure, cross-provider fallback for critical paths. Track which key/region/provider served each request.

Layer 6 — Provisioned throughput for predictable high-scale

Chapter 8's math. Only after you have 60%+ sustained utilization at the top on-demand tier. PT/PTU commits are expensive to walk away from.

What this looks like in practice

A well-architected AI application at scale:

• Every request goes through a rate-limit-aware client that tracks per-provider budgets from response headers
• Client-side rate limiter caps outgoing rate at 85% of provider limits
• On 429, exponential backoff with full jitter and retry-after respect, capped at 5 retries
• Circuit breaker per provider trips on 5+ 5xx or 10+ 429s in 60 seconds
• Cross-provider fallback for critical user-facing paths (Sonnet → GPT-5.6 fallback)
• Instrumentation logs which provider/model actually served each request
• Dashboards alert on drift in fallback ratios, cost per outcome, and cache hit ratios

None of this is glamorous. All of it is what separates AI applications that scale gracefully from ones that page engineers weekly. If you're building an AI product that will need to serve real traffic, invest in these layers early. Retrofitting them under load is significantly harder than getting them right in the first design pass.

Related deep dives

• Cost Optimization Playbook — reduces rate limit pressure by reducing tokens per request
• The AI Errors Nobody Talks About — silent failures adjacent to rate limits
• Gateway comparison — the gateways that implement multi-provider fallback

Frequently asked questions

Fifteen questions that come up repeatedly in reader emails about rate limits. Skim for what applies.

Rate-limit error pages by provider

Every rate-limit error page we've documented, grouped by provider and use case. Bookmark this section.