Self-hosted LLM gateway · Routing

LiteLLM

LiteLLM is a useful reference for teams that want a self-hosted model gateway with provider normalization, model aliases and routing controls owned by the engineering organization.

Third-party guideReviewed 2026-09-26http://localhost:4000

Independent guide This is a third-party tool; these guides are not DeveloperHA products. Features, providers, pricing and security defaults can change; verify the official documentation before installing or sending private data.

What it does

A gateway pattern for presenting multiple model providers behind a compatible API with routing, aliases, retries and operational controls.

Applications call the LiteLLM proxy using a compatible API and an approved model alias.

Applications call the LiteLLM proxy using a compatible API and an approved model alias.

The proxy resolves the alias to one or more configured provider deployments.

The proxy resolves the alias to one or more configured provider deployments.

Routing, retries, fallbacks and concurrency policies run at the gateway boundary.

Routing, retries, fallbacks and concurrency policies run at the gateway boundary.

The gateway becomes an infrastructure service with its own authentication, secrets, observability and upgrades.

The gateway becomes an infrastructure service with its own authentication, secrets, observability and upgrades.

Who it may fit

Choose based on the workflow and trust boundary, not only the feature list.

Good fit for

Teams needing one internal model endpoint
Model aliases and provider routing
Central budgets, retries and operational policy

Verify before using

A self-hosted gateway is production infrastructure, not only a local script.
Secure the proxy, database, logs and provider keys separately.
Test retries and fallbacks against side effects, streaming, tool calls and structured output.

Setup outline

Use this as a mental checklist, then follow the official documentation for the current install and configuration details.

Read the official proxy and routing documentation.

Configure provider keys through environment variables or a secret manager.

Expose only the endpoints and model aliases the application needs.

Add authentication, TLS, rate limits, logs with redaction and health checks before sharing it with a team.

Illustrative endpoint or configuration

model_list:
  - model_name: coding-fast
    litellm_params:
      model: openai/your-model
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2

Data and security reminder A local endpoint does not prove that prompts stay local. Check upstream provider routes, request logs, retention, model training policy, secret storage, network exposure and fallback behavior before using private code or customer data.

Official sources

These links are the authority for current installation, capabilities, releases and security notices.

Official LiteLLM proxy docs ↗Official routing docs ↗LiteLLM documentation ↗

This editorial page was reviewed on 2026-09-26. Project details may change between reviews.

Related AI learning

Related AI tools

Want to understand the engineering trade-offs?

Use the AI learning paths for fundamentals, prompt design, agents, routing and safe development workflows.

Open AI learning See packages