6. Calling models

Available surfaces

Discovering models

curl "$BASE/v1/models" -H "Authorization: Bearer $KEY"

Returns the OpenAI-shaped list of everything you can call — platform models plus any you registered yourself, plus any dedicated pods running in your tenant. If a model answers /v1/chat/completions, it appears here.

For richer detail:

• GET /v1/catalog/models — the full browsable catalog (3,500+ rows, 100 per page, ?limit= up to 500, ?offset=). Each row carries list pricing, the deployability tier, and what you resolve to: bound (can this key call it now), provider, upstream_model, kind (which surface serves it — openai_chat, openai_embed, openai_transcription, openai_image, …) and wire_capabilities, the same booleans the capabilities door answers. Those four are null when bound is false. Filters, all AND-ed: ?provider=openai, ?modality=embedding, ?q=qwen (substring of id or display name), ?min_context=128000, ?max_input_cents_per_mtok=100, ?bound=true, ?deployability=callable. A picker needs one call: GET /v1/catalog/models?bound=true.
• GET /v1/models/{id}/capabilities — the exact parameter support the router enforces for that model, answered for its binding kind: an embedding or image model does not advertise tools, vision or reasoning, and a transcription model advertises only audio_in, timestamps and translation. The answer names the kind.
• GET /v1/models/{id}/health — what that model's serving lane did last: status is healthy, degraded, dead or unknown, beside last_success_at, last_failure_at, last_failure_code and consecutive_failures. Derived from every call the gateway forwards, so a lane whose provider stopped answering reads dead without anyone probing it. Only the lane's own failures count (unreachable, timed out, refused our credential, 5xx); your own 400s, rate limits and budget refusals do not.

Capabilities are refused by name, not silently dropped

If you send a parameter a provider does not support, Nozzle refuses with a typed error naming the parameter — rather than stripping it and returning a plausible answer computed under different settings.

{"error": {
  "code": "request.unsupported_parameter",
  "message": "provider does not support these parameters",
  "details": {"provider": "cerebras", "model": "gemma-4-31b",
              "unsupported_parameters": ["messages[].content.image_url"]}}}

This matters most for prompt caching: silently dropping a cache_control marker costs you the entire cache discount with no way to notice.

Transcription

Send multipart: a file part plus model, and optionally language, prompt, temperature, response_format and timestamp_granularities[].

curl -X POST "$BASE/v1/audio/transcriptions" -H "Authorization: Bearer $KEY" \
  -F file=@meeting.m4a -F model=whisper-1 -F response_format=srt

Uploads are capped at 25 MB, refused with request.payload_too_large carrying details.limit_bytes. Containers: flac, m4a, mp3, mp4, mpeg, mpga, ogg, wav, webm.

Nozzle renders every output format itself from one upstream answer, so srt and vtt are byte-identical in structure whichever backend served you — a hosted provider, our shared pod, or a model you registered. verbose_json relays the provider's own object so its extras survive.

Not every model can time a transcript. OpenAI's gpt-4o-transcribe family and gpt-transcribe answer only json and text; whisper-1 also serves verbose_json, srt, vtt, word timestamps and translations. Ask a model before you send:

curl "$BASE/v1/models/gpt-4o-transcribe/capabilities" -H "Authorization: Bearer $KEY"
# → "capabilities": { …, "timestamps": false, "translation": false }

A format a model cannot produce is refused by name with request.unsupported_parameter and details.unsupported_parameters: ["response_format.srt"] — never an empty subtitle file. The same applies to timestamp_granularities and to sending a json-only model to /v1/audio/translations.

Billing is per minute of audio, on every response format including text. The length comes from the provider when it reports one, and otherwise from the upload itself, which Nozzle measures. A model that reports no length and an upload that cannot be measured is refused (request.invalid, parameter: file) rather than served free. X-Nozzle-Cost-Micro-Cents is on the response, and it reconciles exactly against GET /v1/billing/costs.

Streaming

Set "stream": true. Frames relay byte-for-byte from the provider, so your SDK parses exactly what that provider produced. Add "stream_options": {"include_usage": true} for a terminal usage frame.

Disconnecting aborts the upstream request — the model stops generating and stops costing money.

Vision, embeddings and reasoning models

• Vision is image_url parts (a data: URI or an https URL) on gpt-4.1-mini, gemini-2.5-flash and every Claude model; through the Anthropic drop-in the same image arrives as a base64 source block. Cerebras models refuse it by name. GET /v1/models/{id}/capabilities reports vision per model, honouring provider overrides.
• Embeddings are text-embedding-3-small only: 1536 dimensions, 8191-token input. The dimensions parameter is silently ignored today — a request for 256 returns a 1536-vector.
• Reasoning models (cerebras/qwen-3.8-27b, cerebras/gpt-oss-120b, the gpt-5.x family, Claude with thinking) spend max_tokens on reasoning first. Give them at least 300 or you get an empty 200 with finish_reason: length that still bills. Omit the ceiling entirely and the model's own max_output_tokens from the catalog row is used, not an SDK's hidden 4096.
• gpt-5.x with tools: set reasoning_effort explicitly on /v1/chat/completions or the upstream refuses; tools and reasoning together live on /v1/responses (below).

Prompt caching

Anthropic cache_control markers survive only through the drop-in (/anthropic/v1/messages): a 5.4k-token system block wrote cache_creation_input_tokens: 5411 on the first call and read cache_read_input_tokens: 5411 on the second, buffered and streamed, incremental across a tool loop (measured 2026-09-10 and 2026-09-17). OpenAI and Cerebras cache automatically with no marker and report cached_tokens. The catalog publishes the cache rates beside input and output.

Using the litellm SDK

import litellm
litellm.drop_params = True          # the drop-in denies unknown fields; pin your SDK version
BASE = "https://api.opennozzle.com"

# Claude: the /anthropic base is the only path where cache_control survives
litellm.completion(model="anthropic/claude-haiku-4-5", api_base=f"{BASE}/anthropic",
                   api_key="pk_live_…", messages=[...])

# everything else, OpenAI-shaped
litellm.completion(model="openai/gpt-4.1-mini", api_base=f"{BASE}/v1",
                   api_key="pk_live_…", messages=[...])

The anthropic/ and openai/ prefixes above are litellm's, not Nozzle's. They tell the SDK which dialect to speak, and the SDK strips them before the request leaves your process. Nozzle resolves the model name verbatim, so a raw call must send the plain id: {"model": "claude-haiku-4-5"} to POST /anthropic/v1/messages. Sending anthropic/claude-haiku-4-5 on the wire is 404 resource.not_found — there is no binding by that name.

The cost headers surface as response._hidden_params["additional_headers"]["llm_provider-x-nozzle-cost-micro-cents"]; the unprefixed name returns None.

The Responses API

POST /v1/responses serves OpenAI's Responses grammar — input items, instructions, function tools, reasoning: {effort}, previous_response_id, max_output_tokens — relayed verbatim to the model's upstream. It exists because OpenAI's reasoning models (the gpt-5.x family) refuse function tools together with reasoning_effort on /v1/chat/completions and serve both only here, so an agent that wants tools and reasoning has no other door.

curl "$BASE/v1/responses" -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" -d '{
  "model": "gpt-5.6-luna",
  "input": "What is the weather in Paris? Use the tool.",
  "reasoning": {"effort": "low"},
  "tools": [{"type": "function", "name": "get_weather",
             "parameters": {"type": "object",
                            "properties": {"city": {"type": "string"}},
                            "required": ["city"]}}]
}'

What is Nozzle's and what is not:

• Which models answer. Any chat-kind model whose provider serves /v1/responses: OpenAI proper by default, or any provider row an operator marked responses: true. Ask GET /v1/models/{id}/capabilities — the responses axis is the same fact the router registers on. A model on a provider that does not serve it is refused as request.unsupported_parameter on model, naming /v1/chat/completions.
• Two edits, nothing else. model is rewritten to the upstream's own name and the binding's temperature / top_p / max_tokens overrides are applied in the Responses spelling (max_tokens → max_output_tokens). Everything else in the body reaches the upstream as you wrote it, and the reply comes back as the upstream produced it.
• Streaming. "stream": true returns text/event-stream in the Responses event vocabulary: every event is named by its type (response.created, response.output_text.delta, response.function_call_arguments.delta, …) and the stream ends at response.completed, which carries usage. There is no [DONE] sentinel — the grammar does not define one.
• No fallbacks. A Responses call is stateful across turns (previous_response_id, reasoning items) and bound to the upstream that minted that state, so inline fallbacks are refused by name rather than ignored.
• Billing is identical to chat: usage.input_tokens / usage.output_tokens (output includes reasoning) at the model's chat rates, with the cached-input discount from input_tokens_details.cached_tokens. A buffered call carries X-Nozzle-Cost-Micro-Cents; a streamed one reconciles by X-Request-Id against GET /v1/billing/costs, where the row's kind is inference.responses.