OpenAI tokens_per_min exceeded
What you see when this happens
- 429 with error message specifically mentioning 'tokens per minute'
- `x-ratelimit-remaining-tokens: 0` while `-remaining-requests` shows headroom
- TPM limit hit at low RPM (e.g., 20 requests but 800K tokens)
- Large-context workloads triggering more often than small ones
Common causes
GPT-5.6 with 512K context lets you send massive prompts — TPM drains fast.
Multi-turn conversations without compression grow linearly with token consumption.
Verbose function definitions on every request bloat input tokens.
Without caching, stable system prompts are recharged against TPM every request.
How to fix it
Enable prompt caching
OpenAI automatic prompt caching kicks in for prompts over 1024 tokens. Cached tokens count at 50% against TPM — free savings.
# Structure prompt for cache friendliness
# Stable content first (system + tools), dynamic last (user)
messages = [
{"role": "system", "content": LARGE_STABLE_SYSTEM_PROMPT},
# Cached automatically if this prefix appears identically in prior requests
{"role": "user", "content": user_message},
]
Compress conversation history
Sliding window (keep last N turns) or rolling summary (compress older turns into a summary) — see G5 for the patterns.
def sliding_window(history, keep_turns=8):
system = [m for m in history if m["role"] == "system"]
conv = [m for m in history if m["role"] != "system"]
return system + conv[-keep_turns * 2:] # user+assistant pairs
Route smaller turns to Mini
GPT-5.6 Mini has its own TPM quota. Simple classification or extraction turns can use Mini, saving flagship TPM for complex reasoning.
Batch non-urgent work
Batch API has separate TPM quota and 50% discount. Move background summarization, indexing, and analysis to Batch.
How to prevent it
- Monitor TPM utilization separately from RPM — track headers on every response
- Set alerts at 70% TPM utilization
- Design prompts for cacheability from day one — stable prefix first
- Estimate tokens per request in dashboards, not just request count
Related errors
Deeper reading
Frequently asked questions
Why is TPM hit before RPM?
Modern models have massive context windows. Ten requests with 100K token contexts = 1M TPM burned. Even at 10 RPM (way under any tier's limit), you can hit TPM. This is why token-aware rate limiting matters.
Does prompt caching count toward TPM?
Cached tokens count at reduced rate (~50% on OpenAI). Aggressive caching extends effective TPM 2-5x for cache-friendly workloads. Design your prompts for cache hit — see G8 for schema patterns.