Fallback Strategies: Claude → GPT → Gemini

Why fallback is the hardest reliability decision

Retries survive transient failures. Circuit breakers protect against sustained degradation. Fallback is what you do next — when the breaker opened for your primary provider and you have to serve the user something.

The reason fallback is hard is not implementation. Wiring up a second HTTP client is trivial. The hard part is that the second provider is not the same as the first. Different prompt formats. Different tool schemas. Different structured output mechanisms. Different tokenizers, costs, latencies, quality distributions. Different content moderation policies. A prompt that produces exactly the answer your product needs on Claude Opus may produce something noticeably different on GPT-5.6 — more verbose, less accurate on your domain, or refused entirely.

Every team that ships multi-provider fallback discovers the same three truths:

The scope of this guide

This guide covers cross-provider fallback specifically — Claude falling back to GPT falling back to Gemini, and the equivalents. It does not cover intra-provider fallback (same model, different region) — that's simpler and covered in G1's chapter on region routing. And it does not cover model downgrade within one provider (Sonnet to Haiku) except to note when it's the right tool.

What you'll get

A working implementation that translates prompts across the three major provider families, handles tool use portably, degrades gracefully when structured output schemas differ, and measures quality across tiers so you know when your fallback is silently hurting the product. The complete Python implementation lives in Chapter 12.

Every code sample has been tested in production against the actual providers. Where a translation is lossy (Claude tool use to Gemini function calling, for example), we say so explicitly rather than pretending otherwise.

The four fallback tiers — when to use each

Not all fallbacks are equal. Ranked by preservation of quality and cost, there are four distinct tiers. Every production system should have a preference order across them.

Tier 1: Same model, different region or surface

Anthropic Claude direct falling back to Claude on Bedrock. Vertex Gemini in us-central1 falling back to europe-west4. This is the ideal fallback: identical model, identical quality, only network path and quota bucket differ.

Use when: the primary path is degraded but the underlying model is fine. Almost always the first thing to try. Cost impact: negligible. Quality impact: zero.

Tier 2: Same-quality cross-provider

Claude Sonnet on Anthropic falling back to GPT-5.6 on OpenAI. Two frontier-tier models with comparable capabilities. Your prompt must be portable (Chapter 3), and quality will vary by task — but the tier is close enough that most workloads survive without visible degradation.

Use when: Tier 1 also fails, OR the primary provider is having a sustained outage. Cost impact: moderate (per-token prices differ 10-30% across providers). Quality impact: measurable but often acceptable.

Tier 3: Downgraded model within primary provider

Claude Sonnet falling back to Claude Haiku. GPT-5.6 falling back to GPT-5.6-mini. Same provider, smaller model. Cheaper, faster, less capable. Quality delta is real and depends heavily on task.

Use when: cross-provider fallback also fails, or your primary provider is up but you want to shed load. Cost impact: significant savings (Haiku is ~1/12 the cost of Opus). Quality impact: measurable, often user-visible on complex tasks.

Tier 4: Non-AI response

Cached previous responses, rule-based fallback, or a degraded UX ("AI is temporarily unavailable"). Ultimate protection against all-AI-tier failure.

Use when: all AI tiers fail, or when the request is genuinely uneconomical to serve with AI. Cost impact: none. Quality impact: obvious to users, but preserves service.

The recommended default chain

class FallbackChain:
    tiers = [
        # Tier 1: same model, different surface
        {"provider": "anthropic", "model": "claude-sonnet-4-6"},
        {"provider": "bedrock", "region": "us-east-1", "model": "claude-sonnet-4-6"},
        # Tier 2: cross-provider frontier
        {"provider": "openai", "model": "gpt-5-6"},
        # Tier 3: downgrade
        {"provider": "anthropic", "model": "claude-haiku-4-5"},
        # Tier 4: cached/rules
        {"provider": "cache", "strategy": "last-successful"},
    ]

Chapters 3 through 7 cover the specific translation work required to move between tiers. Chapter 12 shows the complete implementation.

Prompt portability — the message format problem

The first translation problem: the message array format itself. Claude, OpenAI, and Gemini use similar-looking but non-identical formats. Naive copy-paste breaks.

The three formats side by side

# Claude (Anthropic)
{
    "system": "You are a helpful assistant.",
    "messages": [
        {"role": "user", "content": "Hello"},
        {"role": "assistant", "content": "Hi there!"},
        {"role": "user", "content": "How are you?"}
    ]
}

# OpenAI Chat Completions
{
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello"},
        {"role": "assistant", "content": "Hi there!"},
        {"role": "user", "content": "How are you?"}
    ]
}

# Google Gemini
{
    "systemInstruction": {"parts": [{"text": "You are a helpful assistant."}]},
    "contents": [
        {"role": "user", "parts": [{"text": "Hello"}]},
        {"role": "model", "parts": [{"text": "Hi there!"}]},
        {"role": "user", "parts": [{"text": "How are you?"}]}
    ]
}

The four structural differences

1. System prompt placement

Claude puts system prompt at the top level of the request, outside messages. OpenAI puts it as the first message with role="system". Gemini uses systemInstruction, separate from contents.

2. Role names

User and assistant on Claude and OpenAI; user and model on Gemini. Sending role="assistant" to Gemini fails.

3. Content shape

Claude and OpenAI use content as a string or a list of content blocks. Gemini wraps content in parts, each with type-specific fields (text, inlineData, functionCall).

4. Message ordering rules

Claude requires alternating user/assistant roles — two consecutive user messages fail. OpenAI is more permissive. Gemini follows Claude's alternation rule.

The provider-neutral internal format

The cleanest approach is to store prompts in a provider-neutral format internally, then translate at the call site. This isolates the translation logic to one place and makes fallback trivial.

from dataclasses import dataclass, field
from typing import List, Optional, Literal

@dataclass
class NeutralMessage:
    role: Literal["user", "assistant"]
    content: str

@dataclass
class NeutralPrompt:
    system: Optional[str] = None
    messages: List[NeutralMessage] = field(default_factory=list)
    max_tokens: int = 2048
    temperature: float = 0.7

def to_claude(prompt: NeutralPrompt) -> dict:
    return {
        "model": "claude-sonnet-4-6",
        "system": prompt.system or "",
        "messages": [{"role": m.role, "content": m.content} for m in prompt.messages],
        "max_tokens": prompt.max_tokens,
        "temperature": prompt.temperature,
    }

def to_openai(prompt: NeutralPrompt) -> dict:
    msgs = []
    if prompt.system:
        msgs.append({"role": "system", "content": prompt.system})
    msgs.extend({"role": m.role, "content": m.content} for m in prompt.messages)
    return {
        "model": "gpt-5-6",
        "messages": msgs,
        "max_completion_tokens": prompt.max_tokens,
        "temperature": prompt.temperature,
    }

def to_gemini(prompt: NeutralPrompt) -> dict:
    contents = []
    for m in prompt.messages:
        # Note the role rename: assistant → model
        role = "model" if m.role == "assistant" else m.role
        contents.append({"role": role, "parts": [{"text": m.content}]})
    body = {
        "contents": contents,
        "generationConfig": {
            "maxOutputTokens": prompt.max_tokens,
            "temperature": prompt.temperature,
        }
    }
    if prompt.system:
        body["systemInstruction"] = {"parts": [{"text": prompt.system}]}
    return body

What breaks in translation

• Multimodal content — image, audio, and PDF blocks are shaped very differently across providers. Multi-modal fallback needs custom translators per media type.
• Explicit stop sequences — Claude uses stop_sequences; OpenAI uses stop; Gemini uses stopSequences. Rename in the translator.
• Response format constraints — JSON mode, structured output, and tool use each have provider-specific syntax. Chapters 4 and 5 cover this.

Structured output across providers

Structured output is where cross-provider fallback gets subtle. The three providers use different mechanisms — each with distinct guarantees and failure modes.

The three structured output mechanisms

OpenAI: json_schema with strict mode

OpenAI's newest structured output uses a JSON Schema with strict: true. The schema is enforced at decode time — the model literally cannot produce output that violates the schema. This is the strongest guarantee currently available.

response = openai_client.chat.completions.create(
    model="gpt-5-6",
    messages=[...],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "extraction",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "confidence": {"type": "number"}
                },
                "required": ["name", "confidence"],
                "additionalProperties": False
            }
        }
    }
)

Claude: tool_use as structured output

Claude does not have a native structured output mode. The idiomatic pattern is to define a single tool whose parameters match your desired schema, then force Claude to call it with tool_choice. The tool arguments become your structured output.

response = anthropic_client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[...],
    tools=[{
        "name": "extract",
        "description": "Extract structured data",
        "input_schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "confidence": {"type": "number"}
            },
            "required": ["name", "confidence"]
        }
    }],
    tool_choice={"type": "tool", "name": "extract"}
)

# Result is in the tool_use block
tool_block = next(b for b in response.content if b.type == "tool_use")
structured_output = tool_block.input

Gemini: responseSchema

Gemini uses responseSchema on the generationConfig. The response is a JSON string in the text part, guaranteed to match the schema.

response = gemini_client.models.generate_content(
    model="gemini-3-5-pro",
    contents=[...],
    config={
        "response_mime_type": "application/json",
        "response_schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "confidence": {"type": "number"}
            },
            "required": ["name", "confidence"]
        }
    }
)
structured_output = json.loads(response.text)

Schema translation across providers

The good news: all three accept JSON Schema. The bad news: they accept slightly different subsets. OpenAI strict mode requires additionalProperties: false on every object and every property in required. Claude and Gemini are more permissive but silently ignore some constraints. Details:

• OpenAI strict mode requires all properties in required, and no additionalProperties. Enums, oneOf, anyOf partially supported.
• Claude tool schemas accept anything JSON Schema allows but with soft enforcement — the model tries to follow but may deviate.
• Gemini responseSchema maps to protobuf-flavored JSON Schema. Some string formats (email, date-time) don't work.

A portable structured output helper

from typing import Any, Dict

def structured_call(neutral_prompt, schema, provider="anthropic"):
    if provider == "openai":
        r = openai_client.chat.completions.create(
            **to_openai(neutral_prompt),
            response_format={
                "type": "json_schema",
                "json_schema": {
                    "name": "output",
                    "strict": True,
                    "schema": _openai_ify_schema(schema),
                }
            }
        )
        return json.loads(r.choices[0].message.content)
    
    elif provider == "anthropic":
        claude_req = to_claude(neutral_prompt)
        claude_req["tools"] = [{
            "name": "output",
            "description": "Return structured output",
            "input_schema": schema,
        }]
        claude_req["tool_choice"] = {"type": "tool", "name": "output"}
        r = anthropic_client.messages.create(**claude_req)
        block = next(b for b in r.content if b.type == "tool_use")
        return block.input
    
    elif provider == "gemini":
        r = gemini_client.models.generate_content(
            **to_gemini(neutral_prompt),
            config={"response_mime_type": "application/json", "response_schema": schema}
        )
        return json.loads(r.text)

def _openai_ify_schema(schema):
    # OpenAI strict mode requires additionalProperties: false on all objects
    # and all properties listed in required.
    if isinstance(schema, dict):
        if schema.get("type") == "object":
            schema.setdefault("additionalProperties", False)
            schema["required"] = list(schema.get("properties", {}).keys())
        for v in schema.values():
            if isinstance(v, (dict, list)):
                _openai_ify_schema(v)
    elif isinstance(schema, list):
        for item in schema:
            _openai_ify_schema(item)
    return schema

Tool use portability — the three formats

Tool use / function calling is where cross-provider fallback becomes genuinely painful. The wire format is different, the schema constraints differ, the response shape differs, and the streaming shape differs.

Tool definition shapes

# Claude
{
    "tools": [{
        "name": "get_weather",
        "description": "Get current weather for a city",
        "input_schema": {
            "type": "object",
            "properties": {
                "city": {"type": "string"}
            },
            "required": ["city"]
        }
    }],
    "tool_choice": {"type": "auto"}  # or "any", "tool"
}

# OpenAI
{
    "tools": [{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            },
            "strict": True  # optional, forces schema
        }
    }],
    "tool_choice": "auto"  # or "required", specific name
}

# Gemini
{
    "tools": [{
        "functionDeclarations": [{
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            }
        }]
    }],
    "toolConfig": {"functionCallingConfig": {"mode": "AUTO"}}
}

The differences that bite

1. Nesting

OpenAI wraps under function. Gemini wraps under functionDeclarations. Claude is flat. Different nesting per provider.

2. Schema field name

Claude uses input_schema. OpenAI and Gemini use parameters. Same shape, different name.

3. Choice mode

Claude: {"type": "auto"|"any"|"tool"}. OpenAI: "auto"|"required"|{"type": "function", "function": {"name": "..."}}. Gemini: {"mode": "AUTO"|"ANY"|"NONE"}. Semantics roughly match but syntax differs.

4. Multi-tool results

Claude and OpenAI both allow multiple tool calls in one assistant turn. Gemini historically did not, though newer versions do. Fallback logic that assumed single-tool-per-turn breaks when routing to a multi-tool-capable provider.

Tool result shapes

# Claude — tool_use in assistant, tool_result in next user
[
    {"role": "assistant", "content": [
        {"type": "tool_use", "id": "toolu_abc", "name": "get_weather",
         "input": {"city": "Karachi"}}
    ]},
    {"role": "user", "content": [
        {"type": "tool_result", "tool_use_id": "toolu_abc",
         "content": "35°C, sunny"}
    ]}
]

# OpenAI — tool_calls in assistant, tool role for result
[
    {"role": "assistant", "tool_calls": [
        {"id": "call_abc", "type": "function",
         "function": {"name": "get_weather", "arguments": '{"city":"Karachi"}'}}
    ]},
    {"role": "tool", "tool_call_id": "call_abc", "content": "35°C, sunny"}
]

# Gemini — functionCall in model, functionResponse in user
[
    {"role": "model", "parts": [
        {"functionCall": {"name": "get_weather",
                          "args": {"city": "Karachi"}}}
    ]},
    {"role": "user", "parts": [
        {"functionResponse": {"name": "get_weather",
                              "response": {"content": "35°C, sunny"}}}
    ]}
]

The portable translator

@dataclass
class NeutralTool:
    name: str
    description: str
    parameters: Dict[str, Any]

def tools_to_claude(tools: List[NeutralTool]) -> List[dict]:
    return [{"name": t.name, "description": t.description,
             "input_schema": t.parameters} for t in tools]

def tools_to_openai(tools: List[NeutralTool]) -> List[dict]:
    return [{"type": "function", "function": {
        "name": t.name, "description": t.description,
        "parameters": t.parameters,
    }} for t in tools]

def tools_to_gemini(tools: List[NeutralTool]) -> List[dict]:
    return [{"functionDeclarations": [{
        "name": t.name, "description": t.description,
        "parameters": t.parameters,
    } for t in tools]}]

The intermediate transcript

The messier part is translating the conversation history when a tool call happened. If the transcript was generated by Claude and you're now falling back to OpenAI mid-conversation, you have to rewrite prior tool_use / tool_result blocks into OpenAI's shape. Practical rule: never fall back mid-conversation with tools. Reset the conversation when routing to a new provider, or maintain provider stickiness for the duration of a session.

Streaming compatibility — the hardest fallback case

Cross-provider streaming fallback is the hardest case in this entire guide. The three providers stream in three different formats. Once your client has committed to a streaming response, mid-stream fallback essentially cannot preserve the user's ongoing experience.

The three streaming formats

Claude SSE

Server-Sent Events with typed event fields. Every event has a type naming the event kind (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop). Content arrives in deltas nested by content-block index.

OpenAI SSE

Server-Sent Events with a single data: event type. Each data line is a partial completion object. The completion has a choices array whose delta field carries incremental content. The stream ends with data: [DONE].

Gemini streaming

JSON-encoded events over HTTP chunked transfer. Each chunk is a full GenerateContentResponse object with a partial candidates[].content.parts. No terminal marker — the stream just ends.

The recommended pattern: fallback before streaming starts

The only pattern that works cleanly is to try the primary provider, and if it fails before any bytes are yielded to the user, fall back and start streaming from scratch. Once bytes have gone out, the ship has sailed.

async def stream_with_fallback(neutral_prompt, tiers):
    for i, tier in enumerate(tiers):
        try:
            # Get the stream but don't yield yet — verify it starts
            async_iter = start_stream(tier, neutral_prompt)
            first_chunk = await anext(async_iter)
            # First chunk received — commit to this tier
            yield first_chunk
            async for chunk in async_iter:
                yield chunk
            return
        except Exception as e:
            if not is_retryable(e):
                raise
            if i == len(tiers) - 1:
                raise
            logger.warning(f"Stream tier {i} failed pre-first-chunk: {e}")
            continue

# Above works because if the primary raises BEFORE yielding the first chunk,
# we simply never yielded anything to the user and can retry cleanly.

What if the primary starts streaming and then fails?

You have three imperfect options:

1. Fail the response — return an error. Simple, but the user sees a broken UX.
2. Complete on best-effort — hold the partial response and try to complete it on the fallback provider with the same prompt plus what was already generated. Very lossy; often incoherent.
3. Buffer server-side, deliver client-side — complete the stream on your backend, then replay to the client. Users don't see errors, but you lose the low-first-token-latency win of streaming.

For most services, option 3 is the pragmatic middle ground for user-facing text generation. The added latency (until the full response is buffered) is usually preferable to visible errors or mid-response incoherence.

Uniformizing stream events

If your client consumes streams uniformly, translate each provider's stream to a common event shape before yielding.

@dataclass
class NeutralStreamEvent:
    kind: Literal["text_delta", "tool_call_partial", "done"]
    text: Optional[str] = None
    tool_call: Optional[dict] = None

async def translate_claude_stream(stream):
    async for event in stream:
        if event.type == "content_block_delta":
            if event.delta.type == "text_delta":
                yield NeutralStreamEvent(kind="text_delta", text=event.delta.text)
        elif event.type == "message_stop":
            yield NeutralStreamEvent(kind="done")

async def translate_openai_stream(stream):
    async for chunk in stream:
        delta = chunk.choices[0].delta
        if delta.content:
            yield NeutralStreamEvent(kind="text_delta", text=delta.content)
        if chunk.choices[0].finish_reason:
            yield NeutralStreamEvent(kind="done")

async def translate_gemini_stream(stream):
    async for chunk in stream:
        if chunk.candidates and chunk.candidates[0].content:
            for part in chunk.candidates[0].content.parts:
                if hasattr(part, "text") and part.text:
                    yield NeutralStreamEvent(kind="text_delta", text=part.text)
    yield NeutralStreamEvent(kind="done")

Provider quality equivalence — measuring what you're trading away

Falling back to a different model is always a quality tradeoff. The critical question is not whether quality differs — it does — but by how much for your specific workload. Generic benchmarks tell you almost nothing about that. Measurement is essential.

The four measurement dimensions

1. Task-specific accuracy

Does the model complete your specific task correctly? For classification, are the labels right? For extraction, are the fields correct? For summarization, does the summary preserve key points? This is workload-specific and requires labeled evaluation data.

2. Output format compliance

Does the model produce output in the exact shape you expect? Structured output compliance rate varies significantly across providers. Test schema-fit rate on real inputs.

3. Refusal rate

How often does the model refuse to respond? Different providers have different content policies, and a prompt Claude will happily answer may trigger a refusal on Gemini or vice versa. Measure refusal rate per tier per task category.

4. Latency distribution

Not average latency — p50, p95, p99. Some providers have tighter latency distributions than others. If you're in a latency-sensitive product, the tails matter.

The evaluation harness

import statistics
from dataclasses import dataclass
from typing import List, Dict

@dataclass
class EvalResult:
    provider: str
    task_correct: int
    schema_compliant: int
    refused: int
    total: int
    latencies_ms: List[float]

def evaluate(provider_fn, eval_cases: List[dict], judge_fn) -> EvalResult:
    correct, compliant, refused = 0, 0, 0
    latencies = []
    
    for case in eval_cases:
        start = time.time()
        try:
            response = provider_fn(case["input"])
            latency_ms = (time.time() - start) * 1000
            latencies.append(latency_ms)
            
            # Judge the response
            judgment = judge_fn(case, response)
            if judgment.get("refused"):
                refused += 1
                continue
            if judgment.get("schema_valid"):
                compliant += 1
            if judgment.get("task_correct"):
                correct += 1
        except Exception as e:
            logger.error(f"Eval failed: {e}")
    
    return EvalResult(
        provider=provider_fn.__name__,
        task_correct=correct,
        schema_compliant=compliant,
        refused=refused,
        total=len(eval_cases),
        latencies_ms=latencies,
    )

def compare_tiers(eval_cases, judge_fn):
    tiers = {
        "claude-sonnet": call_claude_sonnet,
        "gpt-5-6": call_gpt,
        "gemini-3-5": call_gemini,
        "claude-haiku": call_claude_haiku,
    }
    for name, fn in tiers.items():
        r = evaluate(fn, eval_cases, judge_fn)
        p95 = statistics.quantiles(r.latencies_ms, n=100)[94]
        print(f"{name}: accuracy={r.task_correct/r.total:.1%} "
              f"compliance={r.schema_compliant/r.total:.1%} "
              f"refused={r.refused/r.total:.1%} "
              f"p95={p95:.0f}ms")

Building the eval set

Pull real production requests from the past 30 days, sample uniformly. Get 200-500 examples. Hand-label a random 50 for ground truth. Use those as your gold set. For the remainder, use an LLM judge with human verification of a subset.

Iterate: when your fallback tier's numbers look bad on the eval, look at the failing cases. Often the issue is prompt portability (Chapter 3) rather than model capability — a small prompt adjustment can close much of the gap.

When quality delta is unacceptable

If your Tier 2 fallback (cross-provider) shows more than a 10-15% task accuracy drop, either fix the prompt or accept that Tier 2 shouldn't be part of your default chain for that workload. Better to fail fast than to silently degrade.

The cost of falling back — economic tradeoffs

Fallback has direct cost consequences that most teams don't account for. Understanding them matters for capacity planning and for choosing which fallback chain makes sense.

Per-token cost across major providers (Sep 2026)

# Approximate per-1M-token pricing (input / output)
COSTS = {
    "claude-opus-5":     {"input": 15.00, "output": 75.00},
    "claude-sonnet-4-6": {"input": 3.00,  "output": 15.00},
    "claude-haiku-4-5":  {"input": 0.80,  "output": 4.00},
    "gpt-5-6":           {"input": 2.50,  "output": 10.00},
    "gpt-5-6-mini":      {"input": 0.15,  "output": 0.60},
    "gemini-3-5-pro":    {"input": 1.25,  "output": 5.00},
    "gemini-3-5-flash":  {"input": 0.075, "output": 0.30},
}
# Prices change quarterly. Confirm before capacity planning.

The three cost impacts of fallback

1. Direct per-request cost

Falling from Sonnet ($3/1M in) to GPT-5.6 ($2.50/1M in) is a small savings. Falling from Sonnet to Haiku is a 4x cost reduction. Falling from Sonnet to Opus (as a bug in your chain, or intentionally for critical requests) is a 5x cost increase.

2. Retry amplification

Every failed request that triggers fallback also cost tokens for the failed attempt. During an outage, you're paying for both the failing primary and the fallback secondary — nearly 2x the normal cost for that traffic.

3. Prompt caching loss

Prompt caching lives per-provider. When you fall back to a different provider, cached tokens don't transfer. The cache-miss rate spikes for the duration of the outage. On heavily cached workloads, the effective cost delta is much larger than raw per-token pricing suggests.

Modeling the total cost of a chain

@dataclass
class ChainCost:
    primary_cost_per_1k: float
    secondary_cost_per_1k: float
    tertiary_cost_per_1k: float
    primary_success_rate: float
    secondary_success_rate: float

def expected_cost_per_request(chain: ChainCost, avg_tokens=1500) -> float:
    tokens_1k = avg_tokens / 1000
    p_primary = chain.primary_success_rate
    p_secondary_given_fail = chain.secondary_success_rate
    p_tertiary = (1 - p_primary) * (1 - p_secondary_given_fail)
    
    cost = (
        # Primary attempted always; costs regardless of outcome
        chain.primary_cost_per_1k * tokens_1k
        # Secondary attempted only if primary failed
        + (1 - p_primary) * chain.secondary_cost_per_1k * tokens_1k
        # Tertiary attempted only if both failed
        + p_tertiary * chain.tertiary_cost_per_1k * tokens_1k
    )
    return cost

# Example: normal operation vs during Anthropic incident
normal = ChainCost(
    primary_cost_per_1k=0.018, secondary_cost_per_1k=0.0125,
    tertiary_cost_per_1k=0.005,
    primary_success_rate=0.995, secondary_success_rate=0.99
)
incident = ChainCost(
    primary_cost_per_1k=0.018, secondary_cost_per_1k=0.0125,
    tertiary_cost_per_1k=0.005,
    primary_success_rate=0.30, secondary_success_rate=0.95
)

print(f"Normal: ${expected_cost_per_request(normal):.4f} per request")
print(f"Incident: ${expected_cost_per_request(incident):.4f} per request")
# Incident cost typically 30-50% higher than normal.

Cost-aware fallback ordering

The default fallback order should generally be same-model different-surface, then cross-provider, then downgrade — the same order as tier definition (Chapter 2). But for cost-sensitive workloads, ordering can differ: if you can accept the quality of downgrade, going straight to Haiku on failure can be significantly cheaper than routing through GPT-5.6.

Multi-provider gateways — LiteLLM, OpenRouter, Portkey

Instead of implementing multi-provider fallback yourself, you can use a gateway service that presents a single API and routes to underlying providers. The three main options for 2026 are LiteLLM (self-hosted), OpenRouter (hosted), and Portkey (enterprise-oriented).

What gateways give you

• One API surface — OpenAI-compatible in most cases. Your code calls one endpoint regardless of underlying provider.
• Built-in fallback — declarative fallback chains defined in gateway config, not application code.
• Provider abstraction — prompt translation happens inside the gateway.
• Unified observability — one place for logs, metrics, cost tracking across all providers.

LiteLLM — the self-hosted option

# LiteLLM config.yaml
model_list:
  - model_name: "smart"  # your alias
    litellm_params:
      model: "anthropic/claude-sonnet-4-6"
      api_key: "os.environ/ANTHROPIC_API_KEY"
    model_info:
      priority: 1
  
  - model_name: "smart"  # same alias, fallback candidate
    litellm_params:
      model: "openai/gpt-5-6"
      api_key: "os.environ/OPENAI_API_KEY"
    model_info:
      priority: 2

router_settings:
  routing_strategy: "least-busy"
  fallbacks: [{"smart": ["gpt-5-6", "gemini-3-5-pro"]}]

# Then call
response = openai_client.chat.completions.create(
    model="smart",  # alias resolves through LiteLLM
    messages=[...]
)

Pros: full control, self-hosted, open source, works with any provider LiteLLM supports (100+). Cons: you run and monitor it. Adds a hop of latency (typically 5-20ms). Bugs in your LiteLLM version affect all traffic.

OpenRouter — the hosted option

# OpenRouter presents an OpenAI-compatible API
openai_client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_KEY"]
)

# You address models by "provider/model" pairs
response = openai_client.chat.completions.create(
    model="anthropic/claude-sonnet-4-6",
    extra_body={
        "route": "fallback",
        "models": [
            "anthropic/claude-sonnet-4-6",
            "openai/gpt-5-6",
            "google/gemini-3-5-pro",
        ]
    },
    messages=[...]
)

Pros: zero infrastructure, one bill for all providers, built-in fallback via the models array, transparent about which upstream served the request. Cons: your traffic goes through OpenRouter's infrastructure. Latency and reliability of OpenRouter itself becomes a dependency.

Portkey — the enterprise option

# Portkey uses a virtual key that encodes routing config
from portkey_ai import Portkey

portkey = Portkey(
    api_key=os.environ["PORTKEY_KEY"],
    virtual_key="production-fallback-chain",  # config defined in dashboard
)
response = portkey.chat.completions.create(
    messages=[...],
    model="claude-sonnet-4-6"
)

Pros: deep observability, guardrails and PII redaction, multi-tenant, unified billing, strong analytics. Cons: higher cost tier. Configuration lives in Portkey dashboard (harder to version-control cleanly).

When to use a gateway vs roll your own

• Use a gateway if you have <5 workloads, want to move fast, don't need custom logic per workload. LiteLLM self-hosted or OpenRouter are both good starting points.
• Roll your own if you have >10 workloads with different fallback requirements, need custom quality-adaptive routing, or have hard reliability requirements that don't tolerate a middleware dependency.

Observability — which tier served the request

Without instrumentation, fallback is invisible. Your primary provider silently degrades, requests silently route to secondary, cost silently rises, quality silently drops — and you don't find out until the quarterly billing review or an angry customer email.

The metrics you need

from prometheus_client import Counter, Histogram, Gauge

tier_used = Counter(
    "ai_fallback_tier_used_total",
    "Which fallback tier served the request",
    ["tier", "primary_provider", "actual_provider", "actual_model"]
)

fallback_reason = Counter(
    "ai_fallback_reason_total",
    "Why fallback occurred (from primary)",
    ["reason"]  # circuit_open, exhausted_retries, network, etc.
)

quality_score = Histogram(
    "ai_response_quality_score",
    "Judged quality of response (0.0-1.0)",
    ["tier", "provider"],
    buckets=[0.5, 0.7, 0.8, 0.9, 0.95, 0.99, 1.0]
)

cost_per_request = Histogram(
    "ai_cost_per_request_usd",
    "Cost in USD per completed request",
    ["tier", "provider"],
    buckets=[0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0]
)

The dashboard row that matters

One row on your service dashboard should show tier distribution over time. Under healthy operation, primary serves >95% of requests. Any sustained shift — primary dropping to 80%, secondary rising to 15% — is a leading indicator of a bigger problem.

Correlating quality with tier

Set up a quality-vs-tier chart. If your judge shows quality is holding steady across tiers, fallback is working well. If quality is dropping when secondary or tertiary tiers serve, you have silent degradation happening — fix the fallback tier before shipping something worse to users.

groups:
- name: fallback_observability
  rules:
  # Primary tier usage dropped significantly
  - alert: PrimaryTierUsageDropped
    expr: |
      rate(ai_fallback_tier_used_total{tier="primary"}[5m])
      / rate(ai_fallback_tier_used_total[5m]) < 0.85
    for: 10m
    labels: {severity: warning}
    annotations:
      summary: "Primary tier serving only {{ $value | humanizePercentage }} of traffic"
  
  # Ultimate fallback usage — all AI tiers failed
  - alert: UltimateFallbackUsed
    expr: rate(ai_fallback_tier_used_total{tier="ultimate"}[5m]) > 0
    for: 2m
    labels: {severity: critical}
    annotations:
      summary: "AI fallback ultimately failing — user experience degraded"
  
  # Quality regression on non-primary tier
  - alert: SecondaryTierQualityRegression
    expr: |
      histogram_quantile(0.5,
        rate(ai_response_quality_score{tier="secondary"}[10m])) < 0.85
    for: 10m
    labels: {severity: warning}
    annotations:
      summary: "Secondary tier median quality below acceptable threshold"

Logging with correlation IDs

Log every request with: correlation ID, requested tier, actual tier served, primary error (if any), response quality signals, cost. When something goes wrong two months later, you can query "show me requests that landed on tertiary tier last Tuesday between 2 and 3 PM."

logger.info("ai_call_complete", extra={
    "correlation_id": request.id,
    "requested_tier": "primary",
    "actual_tier": "secondary",
    "primary_error": "CircuitOpenError",
    "actual_provider": "openai",
    "actual_model": "gpt-5-6",
    "input_tokens": usage.input_tokens,
    "output_tokens": usage.output_tokens,
    "cost_usd": calculate_cost(usage, "gpt-5-6"),
    "latency_ms": elapsed * 1000,
})

Common fallback pitfalls — the silent failures

Every team that ships fallback discovers the same failure modes. Being warned about them in advance saves months of quiet damage.

Pitfall 1: Silent quality regression

Fallback tier serves lower-quality responses. No error. No metric changes. Users notice over weeks, complain via support tickets, engineering blames "the model got worse." Reality: your Tier 2 provider was serving 15% of traffic due to a slow primary degradation, and its quality on your specific workload was 20% worse.

Mitigation: Chapter 7's quality-per-tier evaluation. Chapter 10's quality-per-tier metrics.

Pitfall 2: Cost surprise

Bill arrives at the end of the month, 40% higher than expected. Investigation reveals a primary provider outage lasting 6 hours during peak traffic. Fallback worked correctly, but the fallback tier costs 2.5x more than primary. Nobody was watching that.

Mitigation: Cost-per-tier metric with alerts on absolute spend rate spikes, not just relative.

Pitfall 3: Prompt drift

Your team ships prompt improvements over months, tuned against your primary provider. The fallback prompt was written once, never updated. When fallback fires, users get a stale, inferior prompt.

Mitigation: Version prompts in code, not spread across provider-specific files. Whenever you tune the primary prompt, run the eval on the fallback tier too.

Pitfall 4: Fallback tier that's actually the same provider

Primary: Claude Sonnet via Anthropic direct. Fallback: Claude Sonnet via LiteLLM. LiteLLM is configured to use Anthropic upstream. So when Anthropic is down, both tiers fail together. You have no real fallback.

Mitigation: Audit your fallback chain by upstream, not by name. If Tier 2 points to the same upstream as Tier 1 in any way, it's not a real Tier 2.

Pitfall 5: Untested fallback code

Your primary path runs a million times a day. Your fallback runs once a month. It has been broken for six weeks; nobody noticed. When you need it, it fails silently.

Mitigation: Chaos test your fallback path in staging weekly. Force each tier to fire; verify it works end-to-end. Alert if a tier hasn't served real traffic in 30 days.

Pitfall 6: Content moderation differences

A prompt Claude will answer, GPT will refuse. Or vice versa. Your fallback works technically but the user gets a refusal on the fallback tier. From their perspective, the service is broken.

Mitigation: Track refusal rate per tier. When fallback tier's refusal rate on your workload is significantly higher than primary's, that's a real problem to fix.

Pitfall 7: Multi-modal or tool state that doesn't translate

Mid-conversation with a vision or tool-use context, fallback to a provider with different multi-modal or tool semantics. The conversation state translation is lossy or broken. Users see incoherent responses.

Mitigation: Provider stickiness for multi-turn conversations (from Chapter 5). Fall back only at the start of new conversations, not mid-turn.

Pitfall 8: Ultimate fallback that also calls AI

Your "cached response" fallback tier is actually a call to GPT-5.6-mini. When all providers are down, this also fails. There's no true non-AI ultimate fallback.

Mitigation: The ultimate fallback tier must never call any external service. Rules-based response, static content, cached deterministic reply. Anything else and you don't have an ultimate fallback.

The complete fallback implementation

Every idea in this guide, packaged. This is what you drop into a codebase to have production-ready fallback across Claude, GPT, and Gemini.

import time, logging, json
from dataclasses import dataclass, field
from typing import List, Dict, Any, Optional, Callable, Literal
from enum import Enum

from anthropic import Anthropic
from openai import OpenAI
from google import genai

logger = logging.getLogger(__name__)

# ---- Neutral prompt format ----

@dataclass
class NeutralMessage:
    role: Literal["user", "assistant"]
    content: str

@dataclass
class NeutralPrompt:
    system: Optional[str] = None
    messages: List[NeutralMessage] = field(default_factory=list)
    max_tokens: int = 2048
    temperature: float = 0.7
    tools: Optional[List[dict]] = None  # neutral tool defs
    tool_choice: Optional[str] = None   # "auto" | "any" | tool_name

# ---- Tier definitions ----

@dataclass
class Tier:
    name: str
    provider: str        # "anthropic" | "openai" | "gemini" | "cache"
    model: str
    breaker: Any         # CircuitBreaker instance from G3
    kwargs: dict = field(default_factory=dict)

# ---- Per-provider translators ----

def call_anthropic(client: Anthropic, tier: Tier, prompt: NeutralPrompt):
    req = {
        "model": tier.model,
        "max_tokens": prompt.max_tokens,
        "temperature": prompt.temperature,
        "messages": [{"role": m.role, "content": m.content} for m in prompt.messages],
    }
    if prompt.system:
        req["system"] = prompt.system
    if prompt.tools:
        req["tools"] = [{
            "name": t["name"],
            "description": t.get("description", ""),
            "input_schema": t["parameters"],
        } for t in prompt.tools]
        if prompt.tool_choice == "any":
            req["tool_choice"] = {"type": "any"}
        elif prompt.tool_choice and prompt.tool_choice != "auto":
            req["tool_choice"] = {"type": "tool", "name": prompt.tool_choice}
    
    resp = client.messages.create(**req)
    return {
        "provider": "anthropic",
        "model": tier.model,
        "content": _extract_anthropic_content(resp),
        "usage": {
            "input_tokens": resp.usage.input_tokens,
            "output_tokens": resp.usage.output_tokens,
        }
    }

def _extract_anthropic_content(resp):
    parts = []
    for block in resp.content:
        if block.type == "text":
            parts.append({"type": "text", "text": block.text})
        elif block.type == "tool_use":
            parts.append({
                "type": "tool_use", "name": block.name,
                "input": block.input, "id": block.id
            })
    return parts

def call_openai(client: OpenAI, tier: Tier, prompt: NeutralPrompt):
    msgs = []
    if prompt.system:
        msgs.append({"role": "system", "content": prompt.system})
    msgs.extend({"role": m.role, "content": m.content} for m in prompt.messages)
    
    req = {
        "model": tier.model,
        "messages": msgs,
        "max_completion_tokens": prompt.max_tokens,
        "temperature": prompt.temperature,
    }
    if prompt.tools:
        req["tools"] = [{
            "type": "function",
            "function": {
                "name": t["name"],
                "description": t.get("description", ""),
                "parameters": t["parameters"],
            }
        } for t in prompt.tools]
        if prompt.tool_choice == "any":
            req["tool_choice"] = "required"
        elif prompt.tool_choice and prompt.tool_choice != "auto":
            req["tool_choice"] = {"type": "function",
                                  "function": {"name": prompt.tool_choice}}
    
    resp = client.chat.completions.create(**req)
    choice = resp.choices[0]
    return {
        "provider": "openai",
        "model": tier.model,
        "content": _extract_openai_content(choice.message),
        "usage": {
            "input_tokens": resp.usage.prompt_tokens,
            "output_tokens": resp.usage.completion_tokens,
        }
    }

def _extract_openai_content(msg):
    parts = []
    if msg.content:
        parts.append({"type": "text", "text": msg.content})
    if msg.tool_calls:
        for tc in msg.tool_calls:
            parts.append({
                "type": "tool_use", "id": tc.id, "name": tc.function.name,
                "input": json.loads(tc.function.arguments)
            })
    return parts

def call_gemini(client, tier: Tier, prompt: NeutralPrompt):
    contents = []
    for m in prompt.messages:
        role = "model" if m.role == "assistant" else "user"
        contents.append({"role": role, "parts": [{"text": m.content}]})
    
    config = {
        "maxOutputTokens": prompt.max_tokens,
        "temperature": prompt.temperature,
    }
    if prompt.tools:
        config["tools"] = [{
            "functionDeclarations": [{
                "name": t["name"],
                "description": t.get("description", ""),
                "parameters": t["parameters"],
            } for t in prompt.tools]
        }]
    
    kwargs = {"model": tier.model, "contents": contents, "config": config}
    if prompt.system:
        kwargs["system_instruction"] = prompt.system
    
    resp = client.models.generate_content(**kwargs)
    return {
        "provider": "gemini",
        "model": tier.model,
        "content": _extract_gemini_content(resp),
        "usage": {
            "input_tokens": resp.usage_metadata.prompt_token_count,
            "output_tokens": resp.usage_metadata.candidates_token_count,
        }
    }

def _extract_gemini_content(resp):
    parts = []
    if not resp.candidates:
        return parts
    for part in resp.candidates[0].content.parts:
        if hasattr(part, "text") and part.text:
            parts.append({"type": "text", "text": part.text})
        elif hasattr(part, "function_call") and part.function_call:
            parts.append({
                "type": "tool_use",
                "name": part.function_call.name,
                "input": dict(part.function_call.args),
            })
    return parts

# ---- The fallback engine ----

class FallbackChain:
    def __init__(self, tiers: List[Tier], clients: Dict[str, Any]):
        self.tiers = tiers
        self.clients = clients  # {"anthropic": ..., "openai": ..., "gemini": ...}
        self.provider_callers = {
            "anthropic": call_anthropic,
            "openai": call_openai,
            "gemini": call_gemini,
        }
    
    def call(self, prompt: NeutralPrompt, correlation_id: str = ""):
        for i, tier in enumerate(self.tiers):
            if not tier.breaker.allow_request():
                logger.info(f"Tier {tier.name} breaker open, skipping",
                            extra={"correlation_id": correlation_id})
                continue
            
            client = self.clients[tier.provider]
            caller = self.provider_callers[tier.provider]
            
            try:
                start = time.time()
                result = caller(client, tier, prompt)
                tier.breaker.record_success()
                
                logger.info("fallback_success", extra={
                    "correlation_id": correlation_id,
                    "tier": tier.name,
                    "tier_index": i,
                    "provider": tier.provider,
                    "model": tier.model,
                    "latency_ms": (time.time() - start) * 1000,
                    "usage": result["usage"],
                })
                
                # Metrics emission
                metrics.increment(f"ai.tier.used", tags={
                    "tier": tier.name, "provider": tier.provider,
                    "model": tier.model, "tier_index": str(i),
                })
                
                return result
            except Exception as e:
                logger.warning("fallback_tier_failed", extra={
                    "correlation_id": correlation_id,
                    "tier": tier.name, "tier_index": i,
                    "error": str(e), "error_class": type(e).__name__,
                })
                if is_breaker_signal(e):
                    tier.breaker.record_failure()
                # Continue to next tier
        
        # All tiers failed — ultimate fallback
        logger.critical("all_tiers_failed", extra={"correlation_id": correlation_id})
        metrics.increment("ai.tier.ultimate_fallback")
        return self._ultimate_fallback(prompt)
    
    def _ultimate_fallback(self, prompt: NeutralPrompt):
        return {
            "provider": "ultimate_fallback",
            "model": "static",
            "content": [{"type": "text",
                         "text": "Our AI service is temporarily unavailable. "
                                 "Please try again in a few minutes."}],
            "usage": {"input_tokens": 0, "output_tokens": 0},
            "is_degraded": True,
        }

# ---- Usage ----

from circuit_breaker import CircuitBreaker, CircuitBreakerConfig  # from G3

chain = FallbackChain(
    tiers=[
        Tier(name="claude-sonnet-anthropic",
             provider="anthropic", model="claude-sonnet-4-6",
             breaker=CircuitBreaker(CircuitBreakerConfig(name="claude-sonnet"))),
        Tier(name="gpt-5-6-openai",
             provider="openai", model="gpt-5-6",
             breaker=CircuitBreaker(CircuitBreakerConfig(name="gpt-5-6"))),
        Tier(name="gemini-pro",
             provider="gemini", model="gemini-3-5-pro",
             breaker=CircuitBreaker(CircuitBreakerConfig(name="gemini-pro"))),
        Tier(name="claude-haiku-fallback",
             provider="anthropic", model="claude-haiku-4-5",
             breaker=CircuitBreaker(CircuitBreakerConfig(name="claude-haiku"))),
    ],
    clients={
        "anthropic": Anthropic(),
        "openai": OpenAI(),
        "gemini": genai.Client(),
    }
)

prompt = NeutralPrompt(
    system="You are a helpful assistant.",
    messages=[NeutralMessage(role="user", content="What is 2+2?")],
    max_tokens=100
)

result = chain.call(prompt, correlation_id="req-12345")
print(result)

What this gives you

• Provider-neutral prompt format with translators to all three major providers.
• Portable tool definitions across Claude, OpenAI, and Gemini shapes.
• Circuit breaker integration per tier (from G3).
• Structured logging with correlation IDs at every step.
• Metrics emission for tier-used and ultimate-fallback.
• Ultimate non-AI fallback so the caller always gets a response.

What it doesn't do

• Streaming — add async streaming versions using the translators from Chapter 6.
• Multi-modal — images, PDFs, audio need per-media-type translators.
• Structured output — add the schema-aware helpers from Chapter 4.
• Mid-conversation fallback — explicitly avoided (Chapter 5). Design your app to fall back only at conversation start.

Extending in production

Once this works, the extensions worth adding are: async support, structured output helpers, cost-per-tier metrics, quality-per-tier scoring in the metrics pipeline, chaos test hooks. Each is straightforward given the foundation above.

Frequently asked questions

Library & further reading

Everywhere else on AI Error Hub that touches provider fallback, multi-provider architecture, and reliability.