200 OK arrives carrying nothing, a stream stops halfway
through a tool call. This page is the contract for those cases: what the
caller gets, whether GoModel retries or fails over, and where the event shows
up in your monitoring.
For the knobs that control retries and the circuit breaker, see
Resilience. For picking a backup model, see
Failover.
Schema drift
A provider changing its response shape is the failure that gateways usually stay quiet about, because answering it means having an opinion on what a response should look like. GoModel’s opinion is the OpenAI schema, applied generously: unknown members survive, known members are normalized, and only a response we cannot read at all becomes an error.
Passthrough endpoints opt out of all of this:
/p/{provider}/... forwards
the provider’s bytes with none of GoModel’s opinion applied, so a client that
would rather handle drift itself can.
Successful-looking empty responses
A200 with no usable content is a failure wearing a success’s clothes. The
question is whether it is worth failing over on — and for chat completions,
the answer is yes.
Every row logs a warning naming the provider, model, and reason, and
increments
gomodel_empty_responses_total. Because gomodel_requests_total
counts the delivered rows as successes, that counter is the only place the
non-failing rows appear.
A background or in-progress
/v1/responses call legitimately carries
neither output nor usage, and is never reported as empty.Streaming failures
A stream that has already sent bytes cannot be retried — the client has part of an answer. So GoModel’s rule is to decide before the first byte where it can, and to be explicit about truncation where it cannot.
The wait for the first byte never counts toward the idle timeout, so a slow
reasoning model is safe until the provider sends something. After that, any
byte starts the timer — a keep-alive comment included — and every later byte
restarts it, so the limit bounds the gap between reads rather than the length
of the response. Once that first byte is out the response is committed to
200, which is why a stall from then on can only be reported in-band and
never as a status. Details:
stream completion,
idle timeout,
stream-start failover.
Transport and status failures
The conventional cases, for completeness.
Streaming requests are never retried once dispatched, and passthrough
requests only when replay-safe (
GET, HEAD, OPTIONS, PUT, or any
request carrying an Idempotency-Key).
Catching the quiet failures
The rows above that do not fail over are the ones that need an alert, because by definition nothing else complains:gomodel.client.empty_responses,
where the reason arrives as error.type rather than reason, alongside
gomodel.provider.name and gen_ai.request.model. See
Prometheus metrics and
OpenTelemetry.
The dashboard reads the same signals: a model whose recent requests keep
failing marks its provider Degraded even while model discovery still
succeeds, which is what catches an upstream that lists models fine but fails
real calls.
If GoModel itself is down
Putting a gateway in the request path trades provider risk for gateway risk, so the honest numbers matter more than reassurance. GoModel fails loud, not silent: a dead instance gives you connection refused or a502 from whatever fronts it, never a plausible-looking wrong answer.
It is one static binary with a stateless request path, /health and
/health/ready probes, and a 10-second graceful drain on SIGTERM, so
rolling restarts and multiple replicas behind a load balancer work today.
That drain is a cutoff, not a completion guarantee: it is sized for the
requests that can actually finish in ten seconds, and anything still open
past it — an in-flight stream above all — has its connection cut. For
long-lived traffic, drain connections at the load balancer, keep replicas
overlapping long enough to cover a generation, and let clients retry.
What is per-instance and therefore not shared between replicas: circuit
breaker state, rate-limit counters, request-health windows, and sticky
sessions. The production guide covers what that means
for your topology.
If GoModel is down and you need traffic flowing immediately, point
base_url back at the provider directly — the API is OpenAI-compatible in
both directions. Until it is back you lose routing, failover, and logging —
and, because they run inside GoModel’s request path rather than the
provider’s, its rate limits and budget checks stop gating anything at all.