Production Deployment

This page collects the best practices for running GoModel in production. The Quick Start gets you serving traffic in a minute; this page covers what to decide before that install carries real load. Each section states the default, when the default is fine, and when you should change it. A condensed checklist is at the end.

​

Small enough to run anywhere

GoModel is a single static Go binary with no runtime dependencies. It is built with CGO_ENABLED=0 and ships on distroless/static, so the container has no shell, no package manager, and no libc to patch. Those numbers are from the benchmarks page, measured against other gateways on identical hardware. The practical consequence is that GoModel fits comfortably where size and cold starts are penalized: Cloud Run, Fargate, Container Apps, Fly.io, container Lambdas, or a small VM next to your app. It does not need a dedicated node pool.

​

Choose a storage backend

GoModel persists audit logs, usage records, budgets, virtual models, managed API keys, and other admin state through STORAGE_TYPE. SQLite is the default so GoModel works with zero configuration, and it is a real production option for a single instance with a persistent volume. It removes a network hop and a component to operate. It stops being viable in two cases: more than one replica, and any runtime whose filesystem does not survive a restart. Two SQLite replicas do not share a database, they each get their own, so usage, budgets, and managed API keys diverge silently. A key created on one pod does not exist on the other. SQLite also runs with a single open connection by design, to keep one writer and avoid lock contention. Every audit flush, usage flush, and budget check serializes through it, which is a throughput ceiling under sustained load. For Postgres, POSTGRES_MAX_CONNS defaults to 10. Raise it first if you run high concurrency with budgets or rate limits enabled. The default SQLite path is ./data/gomodel.db when a ./data directory already exists next to the binary, and the OS per-user data directory otherwise. That makes the default path dependent on the working directory, so set SQLITE_PATH explicitly in production. The resolved path is printed at startup as storage configured. See Where does GoModel store its data?.

​

Redis

Redis is optional. It backs the shared model catalog snapshot and the exact response cache. Without it, GoModel uses an in-process cache plus a local file cache, which is correct but not shared between replicas. Add Redis when several replicas should share discovery results and cache hits. Redis does not coordinate rate limits or budgets. It is a cache, not a distributed lock.

​

Provider credentials

GoModel takes provider credentials from environment variables, from config.yaml, or from the dashboard’s Providers page. All three work. They do not carry the same risk. For production, configure providers through environment variables and supply the values from your secret manager. Inject them with AWS Secrets Manager, Google Secret Manager, Azure Key Vault, HashiCorp Vault, or a Kubernetes Secret populated by the External Secrets Operator or the Secrets Store CSI driver. There are two reasons, and the second is the one people miss. Secrets stay out of the database. Provider credentials saved from the dashboard are persisted to the provider_credentials store as plaintext. There is no encryption at rest, and the *** masking is applied on the read path only: it stops a key being displayed back over the admin API, but the stored value is readable to anyone with the database file, a backup, a snapshot, or a read replica. The same is true of MCP server headers. Credentials declared via environment variables or config.yaml are never written to the store at all. Config-declared providers cannot be edited at runtime. A provider declared in env or config.yaml is read-only in the dashboard. A provider stored in the database can be edited by anyone holding a dashboard-capable key, including its base_url. Repointing base_url at an attacker-controlled host silently redirects every prompt and the injected upstream key to it. Declaring providers as code means that change can only arrive through your deploy pipeline. GoModel does not support a <VAR>_FILE convention for reading secrets from mounted files, with the single exception of the Vertex AI SERVICE_ACCOUNT_FILE. If your secret tooling mounts files, materialize them into environment variables in your entrypoint. The dashboard path stays the right choice for development, evaluations, and adding a provider without a redeploy. If you use it in production, treat the database as credential material: encrypt the volume, restrict access, and classify backups accordingly.

​

Other credential handling

• Managed API keys are stored as a SHA-256 hash of a high-entropy random secret, never recoverably. The plaintext is shown once at creation. This is deliberately different from provider credentials.
• Client credential headers are redacted before they are persisted. With LOGGING_LOG_HEADERS=true, Authorization, x-api-key, Cookie, and the other credential headers are replaced with [REDACTED] on the write path, so the audit store never receives them.
• Set GOMODEL_MASTER_KEY. If it is empty and no managed keys exist, every request is allowed through unauthenticated, and /admin/* is added to the auth skip list as a lockout-recovery path. An internet-reachable gateway in that state hands anyone the full admin API.

​

TLS

GoModel does not terminate TLS. It serves plain HTTP and expects a reverse proxy or load balancer in front of it. Run one for any deployment that is not loopback-only: without it, client API keys, managed gateway keys, and the master key all cross the network in cleartext. There is also no CORS middleware, so browser clients on another origin cannot call the gateway directly. Add the headers at your proxy if you need that.

​

Air-gapped and offline deployments

GoModel runs fully air-gapped. There is no telemetry, no phone-home, no update check, and no license check. The admin dashboard is embedded in the binary and its fonts are vendored, so the UI loads no CDN assets. Paired with local model servers such as Ollama, SGLang, or vLLM, the gateway needs no route to the public internet. There is exactly one outbound call that is not to a configured provider: the model metadata registry at MODEL_LIST_URL, which supplies pricing, context windows, and capabilities. It is fetched in a background goroutine at startup and again on each cache refresh. That fetch is best-effort. When it fails the gateway boots normally, /health and /health/ready return 200, provider discovery still works because model lists come from each provider’s own /models endpoint, and requests route normally. What you lose is metadata enrichment. If a Bedrock provider is configured, the AWS SDK may also probe the link-local instance metadata endpoint (169.254.169.254) for credentials. An air-gapped install would not configure Bedrock, but it is worth knowing.

​

Running more than one replica

Request handling is stateless, so replicas need no affinity for ordinary traffic. Several features, however, keep state in process memory and do not coordinate across instances. The one that surprises people is rate limits: they are an in-process control, so three replicas with a 100 rpm rule admit up to 300 rpm in aggregate. Size rules per replica, and use budgets as the durable cross-instance control since they read from the shared database.

​

Config changes do not propagate instantly

Admin-managed configuration is durable in the database but served from an in-memory snapshot on each replica. A change saved in the dashboard updates the replica that handled the request immediately. Other replicas pick it up on their own refresh cadence, and two of them have none.

​

Health probes and shutdown

GoModel exposes liveness and readiness separately, and the binary can probe itself, which works in a distroless image with no shell or curl. Point Kubernetes readiness at /health/ready, not /health. A Helm chart is included in the repository under helm/; check its probe paths and its replicaCount against the storage backend you chose before using it as-is. On SIGTERM the gateway drains in-flight HTTP work for 10 seconds, then finishes teardown within 30 seconds, flushing buffered usage and audit entries on the way out. Neither timeout is configurable. In-flight streamed responses are cut at the 10-second mark, so a long completion in progress during a rolling deploy will be truncated. Set terminationGracePeriodSeconds above 30 (45 is a reasonable choice) so the teardown is not racing SIGKILL.

​

Data retention and disk growth

Audit logging and usage tracking write a row per request, and both sweep hourly. Two further points at volume:

• Usage and audit writes are buffered and dropped, not blocked, when the buffer fills. A burst or a slow database can silently lose usage entries, which means lost cost data and under-charged budgets. The only signal is a usage log buffer full warning in the logs, so alert on it.
• Budget checks query the usage table on every request. Keep USAGE_RETENTION_DAYS tight when budgets are enabled, and prefer Postgres over SQLite, whose single connection makes this the dominant cost.

Retention deletes rows but does not shrink a SQLite file. Vacuum it separately if the file size matters.

​

Observability

• Metrics are off by default. Set METRICS_ENABLED=true to expose Prometheus metrics at METRICS_ENDPOINT (/metrics). The endpoint is unauthenticated, so keep it inside your perimeter. See the Prometheus guide.
• Logs are JSON automatically when stdout is not a TTY, so containers get structured logs with no configuration. Set LOG_FORMAT explicitly to override. LOG_LEVEL defaults to info; an invalid value fails startup.
• gomodel_circuit_breaker_state is the gauge to alert on for provider health. It updates per request, so an idle provider keeps its last observed value.
• Request duration for streaming responses measures time to stream establishment, not total stream duration. Read stream latency panels with that in mind.
• There are no token or cost metrics in Prometheus. Spend and token accounting live in the usage database, the dashboard, and GET /v1/usage. Alert on spend from budgets, not from Prometheus.

​

Hardening notes

• Put the admin surface behind your perimeter. The dashboard and /admin/* are served on the same port as the model API and cannot be moved to a separate listener. Restrict them at the proxy or network level, or disable them with ADMIN_ENDPOINTS_ENABLED=false. The dashboard shell and its static assets skip auth by design; only the data they load is gated.
• Restrict the database file. The SQLite data directory is created 0755 and the database file with default permissions, so on a shared host it can be world-readable. Run GoModel as a dedicated user and tighten permissions, especially if any credentials are stored through the dashboard.
• Keep BODY_SIZE_LIMIT sane. It defaults to 10M and is what stops an oversized body becoming a memory-exhaustion vector. An unparseable value falls back to the default with only a warning, so check the spelling.
• Leave pprof disabled. It is off by default and unauthenticated when on.
• Scope managed API keys. Give each consumer its own key with its own user path, which is also what makes per-consumer rate limits and budgets possible. Grant dashboard_access only where needed.

​

Production checklist

• Set GOMODEL_MASTER_KEY to a strong value.
• Terminate TLS in a proxy in front of the gateway.
• Restrict /admin/*, the dashboard, and /metrics at the network perimeter.
• Use Postgres or MongoDB for more than one replica or any ephemeral filesystem; with SQLite, mount a volume and set SQLITE_PATH.
• Declare providers in environment variables or config.yaml, with values from a secret manager.
• Declare budgets and rate limits as code when running several replicas, and size rate limit rules per replica.
• Decide on LOGGING_LOG_BODIES and set retention windows deliberately.
• Point readiness at /health/ready and set terminationGracePeriodSeconds above 30.
• Enable metrics and alert on gomodel_circuit_breaker_state and on usage log buffer full.
• For air-gapped installs, mirror MODEL_LIST_URL or declare model pricing in config so cost tracking and budgets keep working.