A gateway is an integration and operations choice, not a magic source of free intelligence. Start by deciding who owns the endpoint, what data can leave the machine and what happens when a provider fails.
Work through the path
Open a card when you need it. Marking a lesson complete is optional and stays only in this browser.
1Separate endpoint, provider and model
Your application calls an endpoint; the gateway may route to a provider; the provider serves a model. Those layers can change independently.
An OpenAI-compatible endpoint reduces client changes, but compatibility is not identity: features, tool calling, streaming and structured output can differ by route.
Record the selected model and provider in logs or traces without recording secrets or full sensitive prompts.
Try it
Draw the request path for your current setup from client to model and mark every place a prompt, key or log can exist.
Client base URL: http://localhost:20128/v1 Gateway model alias: coding-fast Provider route: provider-specific model Application contract: chat + streaming + approved tools
2Understand 9Router and OmniRoute as routing layers
9Router and OmniRoute are developer-facing local gateway projects: coding tools point at one local endpoint while the gateway handles provider integration, routing and fallback behavior.
The important concept is the boundary: your CLI or editor speaks to a local gateway, and the gateway decides which upstream account or provider receives the request. Read each project’s current official documentation before installing because features, defaults and provider support change quickly.
Compare them on setup, authentication, routing rules, logs, data path, update process, tool compatibility and how you disable or recover from a bad route—not only on the number of providers listed.
Try it
Install one gateway in a disposable environment, point a non-sensitive test client at it and document the actual request path before using it with private code.
Coding tool → localhost gateway → route / fallback → upstream provider Questions to answer: - Which process stores keys? - Which requests are logged? - What happens on 429? - Can a write-capable tool reach a fallback model?
3Choose local Ollama or a hosted gateway deliberately
Ollama is a local model runtime; hosted aggregators and gateways trade local control for access, convenience or provider breadth.
Local inference can keep prompts on your machine and avoid per-request provider billing, but hardware, model quality, latency, updates and operational work become your responsibility.
A hosted gateway can simplify model access and routing, but prompts and outputs may pass through additional systems. Read retention, training, region and provider policies before sending sensitive data.
Try it
Make a two-column decision table for one workload: privacy, quality, latency, cost, operations and failure recovery.
// Local OpenAI-compatible client
var local = new OpenAI({
baseURL: "http://localhost:11434/v1/",
apiKey: "ollama" // placeholder used by the local API
});
// Hosted gateway: keep the real key on the server
var hosted = new OpenAI({
baseURL: process.env.AI_BASE_URL,
apiKey: process.env.AI_API_KEY
});4Use a self-hosted gateway when you need team controls
LiteLLM is useful to study as a team gateway pattern: one compatible API, model aliases, routing strategies, retries, budgets and provider configuration.
Centralization can simplify client configuration and policy, but it creates a critical service that needs authentication, TLS, secret storage, rate limits, logs and upgrades.
Use model aliases so applications depend on an approved capability name while operators can change a provider route deliberately.
Try it
Put provider keys in the gateway environment or secret store, then test a client against the alias rather than the provider name.
model_list:
- model_name: coding-fast
litellm_params:
model: openai/your-model
api_key: os.environ/OPENAI_API_KEY
router_settings:
routing_strategy: simple-shuffle
num_retries: 25Design fallback chains as product behavior
Fallback is not just “try another model”; it can change quality, context, tool support, privacy and cost.
Define which errors are retryable, which models are equivalent enough, how many attempts are allowed and how the user is informed. Do not retry a write with uncertain side effects without an idempotency strategy.
Keep a model allow-list and budget cap. Test the fallback path with the same structured-output and tool-call evaluations as the primary path.
Try it
Force a rate-limit or timeout in a test environment and record what the user sees and what the logs reveal.
primary: coding-strong fallback-1: coding-fast fallback-2: local-small Retry only: timeout, transient 5xx, provider capacity Do not retry automatically: validation failure, auth failure, unsafe tool request
No lesson matches that phrase.
Related tools
Turn the concept into a working example with the browser-based tools.