Retries, timeouts & cancellation

Set a request deadline

Give every Router call an application-level timeout. Choose a deadline that fits the model and prompt, then show a clear waiting state. A network connection staying open does not guarantee that generation will finish.

Retry only temporary failures

Good retry candidates include 429 responses and selected 503 responses such as no_priced_provider_available or request_capacity. Respect Retry-After when present. no_provider_for_length is not worth an immediate retry: shorten the request, or wait for its 60-second Retry-After. Use exponential backoff, random jitter, and a small attempt limit.

1const retryable = new Set([429, 503]);2
3async function waitBeforeRetry(response: Response, attempt: number) {4  const header = response.headers.get("retry-after");5  const headerSeconds = header !== null && /^\d+$/.test(header)6    ? Number(header)7    : NaN;8  const seconds = Number.isSafeInteger(headerSeconds)9    ? headerSeconds10    : Math.min(2 ** attempt, 8);11  const jitter = Math.random(); // Add at most one second after Retry-After.12  const delayMs = (seconds + jitter) * 1000;13  if (delayMs > 2_147_483_647) {14    throw new Error("Retry-After exceeds the supported wait; stop automatic retries");15  }16  await new Promise((resolve) => setTimeout(resolve, delayMs));17}

Do not retry malformed requests, unsupported parameters, invalid credentials, or an unknown model until you change the request.

Reuse the request ID for a retry

Create one UUID for each logical request and send it as Idempotency-Key. If you lose the response, check GET /v1/credits/requests/{request_id} and retry only with the same UUID and unchanged body. A new UUID authorizes another paid request and can create another charge. Keep automatic retries bounded.

A same-UUID retry never starts a second paid run. Each case below shows what Router returns.

Recover a selected offer

A manual selection never silently falls back. For provider_price_changed, fetch the current offer and review its new price. For provider_offer_unavailable or selected_provider_unavailable, wait where appropriate, choose another eligible offer, or explicitly choose Auto. A self_provider_only result requires another account's provider.

Changing the offer, pricing revision, or routing choice changes the request body and requires a new UUID. If the previous outcome is uncertain, check its receipt first. Keep the same UUID and unchanged body when recovering the original request. See choosing a provider.

Scale deadlines with prompt size

The examples use short deadlines suited to short prompts. Long prompts take much longer before the first word: a provider may need several minutes to read a prompt near a 32K context on modest hardware. Raise your client deadline for long inputs and stream them (a non-streaming request above about 24 KB of message text is rejected with stream_required), and keep the deadline above the time you are willing to wait rather than retrying a request that is still running.

Stop receiving when the user leaves

Abort the HTTP request if the user stops receiving, navigates away, or exceeds your deadline. This stops your local response handling; it does not guarantee cancellation of accepted paid work. Generation and settlement can continue after disconnect, and completed work can incur its normal charge within the accepted request maximum. Accepted provider-reported partial usage can also be charged; token counts are not independently verified. Keep the request UUID and check its receipt before starting another paid request.

1import { randomUUID } from "node:crypto";2
3const controller = new AbortController();4const timeout = setTimeout(() => controller.abort(), 45_000);5const requestId = randomUUID(); // Keep for receipt lookup and same-request retry.6
7try {8  const response = await fetch(`${process.env.ROUTER_BASE_URL}/chat/completions`, {9    method: "POST",10    signal: controller.signal,11    headers: {12      Authorization: `Bearer ${process.env.ROUTER_API_KEY}`,13      "Content-Type": "application/json",14      "Idempotency-Key": requestId,15      "X-Max-Charge-Credits": "5.000000",16    },17    body: JSON.stringify({18      model: "gemma3:1b",19      messages: [{ role: "user", content: "Hello!" }],20    }),21  });22  // Inspect response.ok and the error body before reading a success payload.23} finally {24  clearTimeout(timeout);25}

Use the caller UUID and matching X-Request-Id response header to correlate an attempt with usage history and its credit receipt.