Smart AI Gateway: one API, six providers

The problem

Every app in a growing fleet wanted LLM calls: categorization, summarization, chat, scoring. Wiring each app to a specific provider meant six API keys scattered across env files, no shared rate-limit handling, no cost visibility, and a full app outage every time one provider throttled or went down. Multiply that by every new project and you get integration debt that compounds monthly.

What we built

A self-hosted, OpenAI-compatible gateway that sits between all applications and all providers. Apps speak the standard chat-completions API to one internal endpoint. The gateway handles the rest:

• Provider failover. Requests cascade through a prioritized provider chain (Groq, OpenRouter, Cerebras, Gemini, Cohere, Cloudflare). A throttled or failing provider is skipped automatically, the caller never notices.
• Model tiering. Routes map logical model names to concrete provider models, so an app asks for “fast” or “smart” and the gateway picks the best available flavor. Swapping providers is a config change, not a code change.
• Key management in one place. Provider keys live in a single env file on one host, not sprinkled across every consumer repo.
• Observability. Prometheus scrapes per-provider latency, error rates and token counts; Grafana dashboards show exactly which provider served what and when a fallback chain triggered.

Architecture

nginx front, Python routing core, Docker compose deployment. Consumer apps join the gateway network and call an internal hostname, so LLM traffic never leaves the host network. The public internet only ever sees outbound calls to providers.

Outcome

Six providers, one endpoint, near-zero inference cost by riding free tiers with graceful degradation. Since launch, individual providers have had outages and quota resets; no downstream application noticed any of them. Every new project gets production-grade LLM plumbing by adding two lines to its compose file.