Design multi-provider AI routing for failure, not demos

Separate product intent from provider IDs

Application code should ask for a capability such as “fast support answer” or “high-quality product image,” while configuration maps that intent to a versioned provider route. This keeps migrations out of business logic and makes changes reviewable.

Make fallback explicit

A fallback policy should state which errors qualify, how many attempts are allowed, whether the request is idempotent, and what quality or feature differences are acceptable. Never retry a potentially billable non-idempotent request blindly.

route:
  primary: provider-a/model-x
  fallback: provider-b/model-y
  on: [timeout, unavailable]
  max_attempts: 2
  budget_usd: 0.08

Carry one trace across routes

Generate an application request ID before selecting a provider. Store the chosen route, provider request ID, latency, status, retry reason, reported usage, and cost evidence under that trace. Without this, fallback improves availability while making incidents and billing harder to understand.

Prove route equivalence

Use contract fixtures to check response fields and product acceptance fixtures to check outputs. The multi-provider examples show a small OpenAI-compatible routing surface you can extend.