How Routing Works

Every request passes through up to three routing tiers. The gateway tries each tier in order and returns the first successful response.

The three tiers

Request → Tier 1: Operators → Tier 2: LiteLLM → Tier 3: Direct Provider → Response

Tier 1: Operator routing

Operators run Blueprints — on-chain service definitions like the LLM Inference Blueprint. They stake tokens, serve models, and compete on price, latency, and reputation.

• Selected by scoring algorithm: reputation (40%) + latency (30%) + price (30%)
• Discovered automatically from on-chain Blueprint Service Manager contracts
• Can be pinned by blueprint, service, or operator address
• See Operator Routing for the full Blueprint catalog

When it’s used: Default for auto routing mode, required for SpendAuth (on-chain payments).

When it’s skipped: When ZDR or no-train is requested (operators can’t verify compliance). When routing: "provider" is set explicitly.

Tier 2: LiteLLM proxy

An internal proxy that handles 100+ provider integrations with built-in retries and provider-level fallbacks.

When it’s used: Default for standard requests when no operator is available.

When it’s skipped: When ZDR or no-train is requested (LiteLLM’s downstream routing is not compliance-controllable). When LiteLLM is not configured (LITELLM_URL unset).

Tier 3: Direct provider

The gateway calls the provider API directly using platform credentials (or BYOK credentials).

When it’s used: Fallback when tiers 1 and 2 fail. Only tier used when compliance routing is active.

Always used for: ZDR requests, no-train requests, BYOK with compliance flags.

Compliance mode

When zeroDataRetention or disallowPromptTraining is set:

Request → Tier 3: Direct Provider (verified only) → Response

Tiers 1 and 2 are completely bypassed. The gateway routes only to providers with verified compliance agreements. See Zero Data Retention for the trust model.

Routing control