Add Model
Add a new LLM to a live LiteLLM proxy.
Setup
Ask for these if not already known:
LITELLM_BASE_URL — e.g. https://my-proxy.example.com
LITELLM_API_KEY — proxy admin key
API reference: https://litellm.vercel.app/docs/proxy/model_management
Ask the user
- Public model name — what callers will send in
"model": "..."(e.g.gpt-4o,my-claude,llama3) - Provider — pick from the table below
- Credentials — whatever that provider needs
Provider table
| Provider | litellm_params.model |
Extra params |
|---|---|---|
| OpenAI | openai/gpt-4o |
api_key |
| Azure OpenAI | azure/<deployment-name> |
api_key, api_base, api_version |
| Anthropic | anthropic/claude-3-5-sonnet-20241022 |
api_key |
| AWS Bedrock | bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0 |
AWS creds via env |
| Google Vertex | vertex_ai/gemini-1.5-pro |
vertex_project, vertex_location |
| Ollama | ollama/llama3 |
api_base (e.g. http://localhost:11434) |
| Groq | groq/llama-3.3-70b-versatile |
api_key |
| Together AI | together_ai/meta-llama/Llama-3-70b |
api_key |
| Mistral | mistral/mistral-large-latest |
api_key |
Full list: https://docs.litellm.ai/docs/providers
Run
curl -s -X POST "$BASE/model/new" \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{
"model_name": "<public-name>",
"litellm_params": {
"model": "<provider/deployment>",
"api_key": "<key>",
"api_base": "<base_if_needed>",
"api_version": "<version_if_azure>"
}
}'
Test it
After adding, verify it routes:
curl -s -X POST "$BASE/chat/completions" \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "<public-name>",
"messages": [{"role": "user", "content": "say hi"}],
"max_tokens": 10
}'
Output
Show model_id from the response — needed to update or delete the model later.
Report pass/fail from the test call.