AI API Testing Tool: Bring Your Own LLM

This is lesson 9.16 of Shift-Left API in Depth, the product track of the learning center. To follow the demo you need administrator access to a Shift-Left API instance; the self-hosted option needs the Enterprise edition or the trial.

Who picks the model behind an AI API testing tool?

An AI API testing tool is only as good as the model behind it, and only as safe as where that model runs. So who picks the model, and where do your prompts go?

Lesson 9.15 ran a Test Run Pack from a pipeline, on your terms. The AI behind your tests works the same way: Shift-Left API doesn't ship a model of its own. You bring one.

One setting, your model

One setting decides the model. You pick a provider, a model and your own key, and the AI features use it, from test generation to test repair. Switch providers, and your tests and workflows stay as they are.

Where the prompts go depends on where the model runs:

• A hosted provider, such as OpenAI, Anthropic, Gemini, Azure OpenAI or OpenRouter: you use your own key, and the prompts go to that provider's cloud.
• A self-hosted model, on Ollama, LM Studio or vLLM: the prompts go to your own server, and stay on your network. Self-hosted models come with Enterprise, and with the trial.

And without a model? Tests are planned from your API spec first; the AI only reviews them. With a working key, the AI review adds to them. With the AI Engine off, no key, or a key the provider rejects, you still get the spec's tests, and the summary says why the AI wasn't used. If the provider fails mid-run, the spec-based tests are kept.

Find the AI settings

1. Open the account menu and click Platform settings.
2. Open Integrations, then AI Settings. Only administrators see this page, and it applies to the whole system.
3. Use AI Engine is on: the selected model is used when tests are generated.
4. Scroll down to AI Model and API Configuration. Keys are encrypted when they're stored, and only administrators can view them.

Nineteen choices, hosted or self-hosted

Open the Model Provider list: nineteen choices in one list. Hosted providers come first, from OpenAI, Anthropic and Google Gemini down to Hugging Face, each with your own key. The last one is self-hosted: Ollama, LM Studio or vLLM.

Pick Self-hosted, and the form asks for a base URL: the /v1 address of any OpenAI-compatible server. Quick fill writes the usual local ones:

• LM Studio on port 1234
• vLLM on port 8000
• Ollama on port 11434, the default

With Ollama running, the models you've pulled are listed; until then, pick a preset or type a model's name (in the video no model server runs, so the form says it couldn't reach one). The self-hosted API key is optional: leave it empty for Ollama, or set one if a proxy guards your server. With the model on your own server, the AI's prompts stay on your network. This is the route that sets these AI tools for API testing apart for regulated teams.

Test the connection before you save

Back on a hosted provider (OpenRouter in the video, with OpenAI's GPT-4o), the stored key is never shown in the field. Click Test Connection: it sends a tiny ping to the provider. Connection successful means the provider answered, so the key and the model both work, and Key Status reads Verified. After a change, Save These Settings stores it.

See which model reviewed your tests

Open a project built from the sandbox store's spec, pick an endpoint (in the video, Products → Get a single product by ID) and click Generate tests. Read the summary from the top: first the tests planned from your API spec, then the AI review row, with what the model added, the suggestions filtered out as unsafe, and the model's name. Bring another model, and this row names that one. How many tests the AI adds varies from run to run.

Try it yourself

With a trial and the sandbox: open the account menu, then Platform settings → Integrations → AI Settings. Pick a hosted provider and paste your own key, or, if you run Ollama, pull a model, pick Self-hosted and use its Quick fill. Click Test Connection, then Save These Settings. Then generate tests on an endpoint from https://demo.totalshiftleft.ai/openapi/rest.json, and check that the AI review row names your model.

Recap

• An AI API testing tool should let you choose the model: in Shift-Left API, one setting holds nineteen choices, hosted providers with your own key or a self-hosted model.
• The self-hosted route keeps the prompts on your network; Test Connection makes a real call before you save.
• As an AI powered API testing tool, it still writes tests from your spec when no model is there.

Next, lesson 9.17 covers an MCP server for test case generation, so Claude and Cursor can ask Shift-Left API for tests. Its npm package isn't published yet, so for now that's a roadmap item. For a wider view of the category, read our round-up of AI testing tools.