Multi-Provider Load Balancer for AI APIs — Complete Code

Last Thursday at 2:47 PM UTC, Anthropic had a 40-minute partial outage. Every app I know that ran on Claude alone was down for the duration. Every app running a multi-provider load balancer failed over to GPT-5.6 within 3 seconds and served requests normally. The difference isn't the model — it's the 200 lines of TypeScript that pick which model to use when. This post is that code.

What we're building

A drop-in load balancer that sits between your app and multiple LLM providers. Given a request, it picks a provider based on health, cost, or capacity; sends the request; and falls over to the next provider if the first fails. Providers are pluggable: Anthropic, OpenAI, Gemini, or any custom endpoint.

What we're NOT building: a general-purpose service mesh, a distributed load balancer with cluster coordination, or a hosted routing service like OpenRouter or Portkey. This is a single-process helper, meant to be embedded in your app or backend service. It's the middle ground between "one provider hardcoded" and "operate infrastructure to route requests."

The provider adapter interface

Every provider has a different SDK, different request shape, different error types. The load balancer needs a uniform interface. This adapter is the seam.

// providers/base.ts
export interface Message {
  role: "user" | "assistant" | "system";
  content: string;
}

export interface CompletionRequest {
  model?: string;  // provider-specific, optional
  messages: Message[];
  maxTokens: number;
  temperature?: number;
}

export interface CompletionResponse {
  text: string;
  provider: string;
  model: string;
  inputTokens: number;
  outputTokens: number;
  latencyMs: number;
}

export interface ProviderAdapter {
  name: string;
  call(request: CompletionRequest): Promise<CompletionResponse>;
  // Cost per million tokens (input, output)
  pricing: { input: number; output: number };
}


export class ProviderError extends Error {
  constructor(
    public provider: string,
    public statusCode: number | null,
    public retryable: boolean,
    message: string,
  ) {
    super(message);
    this.name = "ProviderError";
  }
}

The Anthropic adapter

// providers/anthropic.ts
import Anthropic from "@anthropic-ai/sdk";
import { ProviderAdapter, CompletionRequest, CompletionResponse, ProviderError } from "./base";

export class AnthropicAdapter implements ProviderAdapter {
  name = "anthropic";
  pricing = { input: 3, output: 15 };  // Sonnet 4.6 per M tokens
  private client: Anthropic;
  private model: string;
  
  constructor(model = "claude-sonnet-4-6") {
    this.client = new Anthropic({ maxRetries: 0 });  // control at balancer
    this.model = model;
  }
  
  async call(req: CompletionRequest): Promise<CompletionResponse> {
    const start = Date.now();
    try {
      const response = await this.client.messages.create({
        model: req.model ?? this.model,
        max_tokens: req.maxTokens,
        temperature: req.temperature,
        messages: req.messages
          .filter((m) => m.role !== "system")
          .map((m) => ({ role: m.role as "user" | "assistant", content: m.content })),
        system: req.messages.find((m) => m.role === "system")?.content,
      });
      
      const textBlock = response.content.find((b) => b.type === "text");
      const text = textBlock ? (textBlock as any).text : "";
      
      return {
        text, provider: this.name, model: this.model,
        inputTokens: response.usage.input_tokens,
        outputTokens: response.usage.output_tokens,
        latencyMs: Date.now() - start,
      };
    } catch (e: any) {
      throw new ProviderError(
        this.name,
        e.status ?? null,
        isRetryableStatus(e.status),
        e.message ?? "Anthropic call failed",
      );
    }
  }
}


function isRetryableStatus(status: number | undefined): boolean {
  if (!status) return true;  // network error, retry
  return status === 429 || status === 529 || (status >= 500 && status < 600);
}

The OpenAI adapter

// providers/openai.ts
import OpenAI from "openai";
import { ProviderAdapter, CompletionRequest, CompletionResponse, ProviderError } from "./base";

export class OpenAIAdapter implements ProviderAdapter {
  name = "openai";
  pricing = { input: 5, output: 20 };  // GPT-5.6 approx
  private client: OpenAI;
  private model: string;
  
  constructor(model = "gpt-5.6") {
    this.client = new OpenAI({ maxRetries: 0 });
    this.model = model;
  }
  
  async call(req: CompletionRequest): Promise<CompletionResponse> {
    const start = Date.now();
    try {
      const response = await this.client.chat.completions.create({
        model: req.model ?? this.model,
        max_tokens: req.maxTokens,
        temperature: req.temperature,
        messages: req.messages.map((m) => ({ role: m.role, content: m.content })),
      });
      
      return {
        text: response.choices[0].message.content ?? "",
        provider: this.name, model: this.model,
        inputTokens: response.usage?.prompt_tokens ?? 0,
        outputTokens: response.usage?.completion_tokens ?? 0,
        latencyMs: Date.now() - start,
      };
    } catch (e: any) {
      throw new ProviderError(
        this.name,
        e.status ?? null,
        isRetryableStatus(e.status),
        e.message ?? "OpenAI call failed",
      );
    }
  }
}


function isRetryableStatus(status: number | undefined): boolean {
  if (!status) return true;
  return status === 429 || (status >= 500 && status < 600);
}

The Gemini adapter

// providers/gemini.ts
import { GoogleGenAI } from "@google/genai";
import { ProviderAdapter, CompletionRequest, CompletionResponse, ProviderError } from "./base";

export class GeminiAdapter implements ProviderAdapter {
  name = "gemini";
  pricing = { input: 3.5, output: 10.5 };  // Gemini 3.5 Pro
  private client: GoogleGenAI;
  private model: string;
  
  constructor(model = "gemini-3.5-pro") {
    this.client = new GoogleGenAI({ apiKey: process.env.GOOGLE_API_KEY! });
    this.model = model;
  }
  
  async call(req: CompletionRequest): Promise<CompletionResponse> {
    const start = Date.now();
    try {
      const contents = req.messages
        .filter((m) => m.role !== "system")
        .map((m) => ({
          role: m.role === "assistant" ? "model" : "user",
          parts: [{ text: m.content }],
        }));
      
      const systemInstruction = req.messages.find((m) => m.role === "system")?.content;
      
      const response = await this.client.models.generateContent({
        model: this.model,
        contents,
        config: {
          maxOutputTokens: req.maxTokens,
          temperature: req.temperature,
          systemInstruction,
        },
      });
      
      return {
        text: response.text ?? "",
        provider: this.name, model: this.model,
        inputTokens: response.usageMetadata?.promptTokenCount ?? 0,
        outputTokens: response.usageMetadata?.candidatesTokenCount ?? 0,
        latencyMs: Date.now() - start,
      };
    } catch (e: any) {
      const status = e.response?.status ?? null;
      throw new ProviderError(
        this.name,
        status,
        !status || status === 429 || (status >= 500 && status < 600),
        e.message ?? "Gemini call failed",
      );
    }
  }
}

Health tracking

The load balancer needs to know which providers are healthy. A simple sliding-window error rate tracker suffices for most cases.

// health.ts
export class HealthTracker {
  private windows: Map<string, { successes: number; failures: number; lastReset: number }> = new Map();
  private readonly windowMs: number;
  private readonly unhealthyThreshold: number;
  
  constructor(windowMs = 60_000, unhealthyThreshold = 0.5) {
    this.windowMs = windowMs;
    this.unhealthyThreshold = unhealthyThreshold;
  }
  
  private getWindow(provider: string) {
    let w = this.windows.get(provider);
    const now = Date.now();
    if (!w || now - w.lastReset > this.windowMs) {
      w = { successes: 0, failures: 0, lastReset: now };
      this.windows.set(provider, w);
    }
    return w;
  }
  
  recordSuccess(provider: string) {
    this.getWindow(provider).successes++;
  }
  
  recordFailure(provider: string) {
    this.getWindow(provider).failures++;
  }
  
  isHealthy(provider: string): boolean {
    const w = this.getWindow(provider);
    const total = w.successes + w.failures;
    if (total < 10) return true;  // not enough data
    return w.failures / total < this.unhealthyThreshold;
  }
  
  errorRate(provider: string): number {
    const w = this.getWindow(provider);
    const total = w.successes + w.failures;
    return total > 0 ? w.failures / total : 0;
  }
}

The load balancer

Now the core class. Takes a list of provider adapters, picks one per request, handles failures, tracks health, exposes metrics hooks.

// load-balancer.ts
import { ProviderAdapter, CompletionRequest, CompletionResponse, ProviderError } from "./providers/base";
import { HealthTracker } from "./health";

export type Strategy = "priority" | "weighted" | "cost";

export interface LoadBalancerOptions {
  providers: ProviderAdapter[];
  strategy?: Strategy;
  weights?: Record<string, number>;  // for "weighted"
  healthTracker?: HealthTracker;
  onProviderCall?: (provider: string) => void;
  onProviderSuccess?: (provider: string, latencyMs: number) => void;
  onProviderFailure?: (provider: string, error: ProviderError) => void;
  onFailover?: (from: string, to: string) => void;
  onAllFailed?: () => void;
}


export class LoadBalancer {
  private providers: ProviderAdapter[];
  private strategy: Strategy;
  private weights: Record<string, number>;
  private health: HealthTracker;
  private hooks: Required<Pick<LoadBalancerOptions,
    "onProviderCall" | "onProviderSuccess" | "onProviderFailure" | "onFailover" | "onAllFailed"
  >>;
  
  constructor(opts: LoadBalancerOptions) {
    if (opts.providers.length === 0) {
      throw new Error("LoadBalancer requires at least one provider");
    }
    this.providers = opts.providers;
    this.strategy = opts.strategy ?? "priority";
    this.weights = opts.weights ?? {};
    this.health = opts.healthTracker ?? new HealthTracker();
    this.hooks = {
      onProviderCall: opts.onProviderCall ?? (() => {}),
      onProviderSuccess: opts.onProviderSuccess ?? (() => {}),
      onProviderFailure: opts.onProviderFailure ?? (() => {}),
      onFailover: opts.onFailover ?? (() => {}),
      onAllFailed: opts.onAllFailed ?? (() => {}),
    };
  }
  
  async call(request: CompletionRequest): Promise<CompletionResponse> {
    const ordered = this.orderProviders(request);
    const attempted: string[] = [];
    let lastError: ProviderError | null = null;
    
    for (const provider of ordered) {
      attempted.push(provider.name);
      this.hooks.onProviderCall(provider.name);
      
      try {
        const response = await provider.call(request);
        this.health.recordSuccess(provider.name);
        this.hooks.onProviderSuccess(provider.name, response.latencyMs);
        return response;
      } catch (e) {
        const err = e instanceof ProviderError 
          ? e 
          : new ProviderError(provider.name, null, true, String(e));
        this.health.recordFailure(provider.name);
        this.hooks.onProviderFailure(provider.name, err);
        lastError = err;
        
        // Non-retryable: don't fail over
        if (!err.retryable) throw err;
        
        // Log the failover
        const nextIdx = ordered.indexOf(provider) + 1;
        if (nextIdx < ordered.length) {
          this.hooks.onFailover(provider.name, ordered[nextIdx].name);
        }
      }
    }
    
    this.hooks.onAllFailed();
    throw new Error(
      `All ${attempted.length} providers failed. Last error: ${lastError?.message}`
    );
  }
  
  private orderProviders(request: CompletionRequest): ProviderAdapter[] {
    // Filter to healthy providers first
    const healthy = this.providers.filter((p) => this.health.isHealthy(p.name));
    const unhealthy = this.providers.filter((p) => !this.health.isHealthy(p.name));
    // Unhealthy ones go last, in case all healthy fail
    const pool = [...healthy, ...unhealthy];
    
    switch (this.strategy) {
      case "priority":
        return pool;  // as given
      case "weighted":
        return this.weightedShuffle(pool);
      case "cost":
        return this.costOrdered(pool, request);
      default:
        return pool;
    }
  }
  
  private weightedShuffle(pool: ProviderAdapter[]): ProviderAdapter[] {
    // Weighted random by configured weight (default 1)
    const weighted = pool.map((p) => ({
      provider: p,
      weight: this.weights[p.name] ?? 1,
    }));
    const result: ProviderAdapter[] = [];
    const remaining = [...weighted];
    while (remaining.length > 0) {
      const total = remaining.reduce((s, x) => s + x.weight, 0);
      let r = Math.random() * total;
      for (let i = 0; i < remaining.length; i++) {
        r -= remaining[i].weight;
        if (r <= 0) {
          result.push(remaining[i].provider);
          remaining.splice(i, 1);
          break;
        }
      }
    }
    return result;
  }
  
  private costOrdered(pool: ProviderAdapter[], request: CompletionRequest): ProviderAdapter[] {
    // Estimate cost per request; cheapest first
    const inputTokens = request.messages.reduce(
      (s, m) => s + Math.ceil(m.content.length / 4), 0
    );
    const outputTokens = request.maxTokens / 2;  // rough estimate
    
    return [...pool].sort((a, b) => {
      const costA = (inputTokens * a.pricing.input + outputTokens * a.pricing.output) / 1_000_000;
      const costB = (inputTokens * b.pricing.input + outputTokens * b.pricing.output) / 1_000_000;
      return costA - costB;
    });
  }
}

Using it

// app.ts
import { LoadBalancer } from "./load-balancer";
import { AnthropicAdapter } from "./providers/anthropic";
import { OpenAIAdapter } from "./providers/openai";
import { GeminiAdapter } from "./providers/gemini";

const lb = new LoadBalancer({
  providers: [
    new AnthropicAdapter("claude-sonnet-4-6"),
    new OpenAIAdapter("gpt-5.6"),
    new GeminiAdapter("gemini-3.5-pro"),
  ],
  strategy: "priority",  // try Anthropic first, then failover
  onProviderCall: (name) => console.log(`Calling ${name}`),
  onProviderFailure: (name, err) => 
    console.warn(`${name} failed: ${err.message}`),
  onFailover: (from, to) => 
    console.log(`Failing over from ${from} to ${to}`),
});


// Use just like a single-provider call
const response = await lb.call({
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "What's the capital of France?" },
  ],
  maxTokens: 100,
});

console.log(response.text);
console.log(`Served by ${response.provider} (${response.model})`);
console.log(`Latency: ${response.latencyMs}ms`);
console.log(`Tokens: ${response.inputTokens} in, ${response.outputTokens} out`);

Strategy comparison

Three strategies with different tradeoffs:

Priority (default)

Always try providers in configured order. Fail over only when the primary fails. Use when: one provider is clearly your preferred choice and others are backups. Failover happens transparently on outages.

Weighted

Randomly pick per-request based on weights. Higher weight = more often. Use when: you want to distribute load across providers proactively (spreading rate limits) or when you're A/B testing quality.

// 60% Anthropic, 30% OpenAI, 10% Gemini
const lb = new LoadBalancer({
  providers: [...],
  strategy: "weighted",
  weights: { anthropic: 60, openai: 30, gemini: 10 },
});

Cost

Always try the cheapest provider first, failover to more expensive ones. Use when: cost matters more than quality and your workload works on any of the providers. Not appropriate when quality varies significantly between providers.

What this doesn't cover

The 200 lines above cover 80% of production cases. What's missing:

Each of these is worth a post on its own. Together, they're the operational polish layer. The load balancer above is the foundation.

The observability you actually need

Wire the metrics hooks into whatever observability stack you use. The minimum useful set:

import { Counter, Histogram } from "prom-client";

const callCounter = new Counter({
  name: "llm_provider_calls_total",
  help: "Total provider calls",
  labelNames: ["provider"],
});
const successCounter = new Counter({
  name: "llm_provider_success_total",
  help: "Successful provider calls",
  labelNames: ["provider"],
});
const failureCounter = new Counter({
  name: "llm_provider_failure_total",
  help: "Failed provider calls",
  labelNames: ["provider", "status"],
});
const latencyHistogram = new Histogram({
  name: "llm_provider_latency_ms",
  help: "Provider latency",
  labelNames: ["provider"],
  buckets: [50, 100, 250, 500, 1000, 2500, 5000, 10000],
});
const failoverCounter = new Counter({
  name: "llm_failovers_total",
  help: "Failover events",
  labelNames: ["from", "to"],
});


const lb = new LoadBalancer({
  providers: [...],
  onProviderCall: (name) => callCounter.labels(name).inc(),
  onProviderSuccess: (name, latencyMs) => {
    successCounter.labels(name).inc();
    latencyHistogram.labels(name).observe(latencyMs);
  },
  onProviderFailure: (name, err) => 
    failureCounter.labels(name, String(err.statusCode ?? "unknown")).inc(),
  onFailover: (from, to) => failoverCounter.labels(from, to).inc(),
});

With these five metrics, you can answer: which providers are healthy right now, what's my failover rate, which provider is fastest, where are my error concentrations. Dashboards write themselves.

The insurance policy framing

A load balancer isn't free. Each adapter is code you maintain. Each provider is an account, keys to rotate, docs to keep up with. Cross-provider testing is complexity you didn't have when you ran on Claude alone.

The value proposition is that it's insurance. During Anthropic's 40-minute outage in October 2026, apps with load balancers stayed up; apps without them didn't. During OpenAI's tier-4 rate limit reduction earlier in the year, apps with load balancers absorbed the shift by shifting weight; apps without them had to scramble. The complexity cost is upfront; the payoff is when things break, which will happen. Every major provider has had multi-hour outages in the past two years.

The threshold for "worth building" is somewhere around "revenue depends on the app being up." For consumer chatbots serving 10K+ users, yes. For internal tools where a 30-minute outage means "team goes to lunch," probably not. Judge accordingly.

What's next

Two directions from here:

First, harden this: add exponential backoff (G9), circuit breakers (B4), Redis-backed state, streaming support. That gets you to a load balancer that survives real scale.

Second, extend it: add tool-use routing (some providers handle certain tool patterns better), structured output routing (route to whichever provider best supports your schema shape), vision routing (only some providers support images). That gets you to a load balancer that's a strategic asset, not just insurance.

The 200 lines above are the starting point. Everything else is polish on the foundation.