Self-hosted LLM
Generate tests with a model you control
Point AI test generation at Ollama, LM Studio, vLLM, or any OpenAI-compatible endpoint you host. Specs and prompts stay on infrastructure you control — no cloud AI provider is called.
Why teams choose a self-hosted model
Most AI test generation sends your API specification — endpoints, schemas, sometimes example payloads — to a third-party LLM provider. For regulated workloads in banking, healthcare, insurance, and government, that data flow is often a non-starter under security and residency policy, regardless of how strong the provider's own security is. Pointing generation at a model you run yourself removes that question entirely: the spec never leaves the base URL you configure.
Ollama
http://localhost:11434/v1Quick-fill base URL in AI Settings
LM Studio
http://localhost:1234/v1Quick-fill base URL in AI Settings
vLLM
http://localhost:8000/v1Quick-fill base URL in AI Settings
Running your model in Docker? Use http://host.docker.internal:<port>/v1 in place of localhost.
Four steps to connect a self-hosted model
Open AI Settings
Settings → Integrations → AI Settings. You must be an Administrator — non-admins see Access Denied.
Choose Self-hosted as the Model Provider
In the Model Provider dropdown, alongside ~20 cloud providers (OpenAI, Anthropic, Google Gemini, Azure OpenAI, and others), select Self-hosted (Ollama / LM Studio / vLLM). This is an Enterprise-plan capability — on other plans the option shows "— Enterprise only".
Set the Base URL
Use one of the Quick fill buttons for Ollama, LM Studio, or vLLM, or enter any OpenAI-compatible /v1 base URL. Running the model in Docker? Use http://host.docker.internal:<port>/v1 instead of localhost.
Pick a model and test the connection
Click Refresh available models to list what's installed on your server, choose one, then Test Connection — a small ping that verifies the model without saving anything. Save Settings once it succeeds.
Full walkthrough with every field and troubleshooting step: AI Settings documentation.
Not locked into three models
Ollama, LM Studio, and vLLM get one-click quick-fill buttons because they're the most common self-hosted setups, but the underlying field accepts any OpenAI-compatible /v1 base URL. If your infrastructure team runs a different inference server that exposes that same API shape, point the Base URL field at it directly.
Self-hosted LLM is one of ~20 provider families available in AI Settings, alongside cloud options like OpenAI, Anthropic, Google Gemini, and Azure OpenAI — so teams can run cloud models for non-sensitive projects and a self-hosted model for anything that needs to stay inside the perimeter, without switching platforms.
Self-hosted LLM — frequently asked questions
Which self-hosted LLMs does Total Shift Left support?
AI Settings has one-click Quick fill buttons for Ollama, LM Studio, and vLLM, but the Base URL field accepts any OpenAI-compatible /v1 endpoint — so any model server that exposes that API shape works, not only the three with quick-fill buttons.Is self-hosted LLM available on every plan?
No — it's an Enterprise-plan capability. On other plans, selecting it in AI Settings shows "Self-hosted LLM is available on the Enterprise plan only" with an upgrade link.Does anything leave my infrastructure when I use a self-hosted model?
No cloud AI provider is called. Test generation prompts and the API specs behind them go to the base URL you configure — your own Ollama, LM Studio, or vLLM instance — and nowhere else. This is why self-hosted deployment paired with a self-hosted LLM is the option regulated enterprises (BFSI, healthcare, public sector) use for air-gapped environments.What if my model server runs in Docker?
Use http://host.docker.internal:<port>/v1 instead of localhost — from inside a Docker container, localhost refers to the container itself, not the host machine running your model server.Do I need an API key for a self-hosted model?
The Self-hosted API key field is optional — most local model servers (Ollama, LM Studio, vLLM in their default configuration) don't require one. If your setup does enforce a key, add it under Edit All API Keys.What is the embedding model field for?
An optional, collapsed Advanced section lets you set an embedding model (default nomic-embed-text) used only for offline RAG / semantic search when no cloud embedding key is configured — it's not required for basic test generation.How do I know the connection actually works before saving?
Click Test Connection. It sends a small ping to your configured endpoint to verify the base URL and model — nothing is saved until you click Save Settings afterward. A failed test shows an error code so you can diagnose the base URL or model name before committing the change.
More integrations
Keep your specs on your own infrastructure
Enterprise trial mirrors the full self-hosted LLM setup. Or talk to an architect about your deployment topology.