Self-Hosted LLM Integration: Ollama, LM Studio and vLLM
The self-hosted LLM integration points Shift-Left API's AI test generation, gap filling and Fix with AI at a model you run: Ollama, LM Studio, vLLM or any OpenAI-compatible server. Specs and prompts are not sent to a cloud AI provider. Available on Trial and Enterprise; other plans bring their own cloud key or use no AI.
Why teams choose a self-hosted model
Most AI test generation sends your API specification — endpoints, schemas, sometimes example payloads — to a third-party LLM provider. For regulated workloads in banking, healthcare, insurance, and government, that data flow is often a non-starter under security and residency policy, regardless of how strong the provider's own security is. Pointing generation at a model you run yourself removes that question entirely: the spec never leaves the base URL you configure.
Ollama
http://localhost:11434/v1Quick-fill base URL in AI Settings
LM Studio
http://localhost:1234/v1Quick-fill base URL in AI Settings
vLLM
http://localhost:8000/v1Quick-fill base URL in AI Settings
Running your model in Docker? Use http://host.docker.internal:<port>/v1 in place of localhost.
Four steps to connect a self-hosted model
Open AI Settings
Settings → Integrations → AI Settings. You must be an Administrator — non-admins see Access Denied.
Choose Self-hosted as the Model Provider
In the Model Provider dropdown, alongside cloud providers such as OpenAI, Anthropic, Google Gemini and Azure OpenAI, select Self-hosted (Ollama / LM Studio / vLLM). It is available on the Trial and Enterprise plans, not on Professional or the free Citizen Developer edition.
Set the Base URL
Use one of the Quick fill buttons for Ollama, LM Studio, or vLLM, or enter any OpenAI-compatible /v1 base URL. Running the model in Docker? Use http://host.docker.internal:<port>/v1 instead of localhost.
Pick a model and test the connection
Click Refresh available models to list what's installed on your server, choose one, then Test Connection — a small ping that verifies the model without saving anything. Save Settings once it succeeds.
Full walkthrough with every field and troubleshooting step: AI Settings documentation.
Not locked into three models
Ollama, LM Studio, and vLLM get one-click quick-fill buttons because they're the most common self-hosted setups, but the underlying field accepts any OpenAI-compatible /v1 base URL. If your infrastructure team runs a different inference server that exposes that same API shape, point the Base URL field at it directly.
Self-hosted LLM is one of the provider options in AI Settings, alongside cloud options like OpenAI, Anthropic, Google Gemini, and Azure OpenAI — so teams can run cloud models for non-sensitive projects and a self-hosted model for anything that needs to stay inside the perimeter, without switching platforms.
Self-hosted LLM — frequently asked questions
Which self-hosted LLMs does Shift-Left API support?
AI Settings has one-click Quick fill buttons for Ollama, LM Studio, and vLLM, but the Base URL field accepts any OpenAI-compatible /v1 endpoint — so any model server that exposes that API shape works, not only the three with quick-fill buttons.
Which cloud AI providers can I use instead?
On every plan you can bring your own key for any of 19 provider families, including Anthropic Claude, OpenAI, Google Gemini and Azure OpenAI. The current Claude models are supported. Each provider is sent the request format its API accepts; we say supported rather than certified, because not every provider has been tried with a live key.
Is self-hosted LLM available on every plan?
No. It is available on the Trial and Enterprise plans. Professional and the free Citizen Developer edition use a cloud provider with your own API key.
Does anything leave my infrastructure when I use a self-hosted model?
No cloud AI provider is called. Test generation prompts and the API specs behind them go to the base URL you configure — your own Ollama, LM Studio, or vLLM instance — and nowhere else. Paired with a self-hosted install, this keeps AI traffic inside your network. Licence validation and updates still contact Total Shift Left unless arranged otherwise, so ask us about offline licensing for air-gapped installs.
What if my model server runs in Docker?
Use http://host.docker.internal:<port>/v1 instead of localhost — from inside a Docker container, localhost refers to the container itself, not the host machine running your model server.
Do I need an API key for a self-hosted model?
The Self-hosted API key field is optional — most local model servers (Ollama, LM Studio, vLLM in their default configuration) don't require one. If your setup does enforce a key, add it under Edit All API Keys.
What is the embedding model field for?
An optional, collapsed Advanced section lets you set an embedding model (default nomic-embed-text) used only for offline RAG / semantic search when no cloud embedding key is configured — it's not required for basic test generation.
How do I know the connection actually works before saving?
Click Test Connection. It sends a small ping to your configured endpoint to verify the base URL and model — nothing is saved until you click Save Settings afterward. A failed test shows an error code so you can diagnose the base URL or model name before committing the change.
Keep your specs on your own infrastructure
The 15-day trial includes the self-hosted LLM option. Or talk to an engineer about your deployment topology and AI policy.