Chat completions

Overview

Send an OpenAI Chat Completions request to /v1/chat/completions. Add an optional inferada object when the request needs processors or gateway-specific routing. Standard response fields remain unchanged. Operational metadata appears under the top-level inferada object, while billing metadata appears under usage.inferada. Unknown OpenAI request fields pass through to the model.

Compatibility and Inferada extensions

Inferada additions do not replace standard fields. Clients that ignore unknown JSON fields can continue to use the OpenAI-compatible response. Chat Completions and Responses place Inferada metadata in the same locations for buffered responses and terminal stream events.

inferadaobject

Top-level operational metadata: request_id, timings, routing, processors, tools, buffering and guard outcomes.

usage.inferadaobject

Billing-only metadata. Currently contains cost_eur; operational fields never appear here.

x-request-idheader

Stable request correlation for every transport, including binary audio and file responses.

The @inferada-ai/ai-sdk-provider package maps these additions to providerMetadata.inferada in the Vercel AI SDK. Use its .passthrough provider when application code should receive only standard AI SDK result fields. The raw client also exposes client.passthrough, which removes Inferada metadata from OpenAI-shaped JSON and SSE data. Transport headers remain available in both modes.

import { createInferada } from '@inferada-ai/ai-sdk-provider'
import { generateText } from 'ai'

const inferada = createInferada({ apiKey: process.env.INFERADA_API_KEY })

// Enriched: result.providerMetadata?.inferada
const enriched = await generateText({ model: inferada('your-model-id'), prompt: 'Hello' })

// Clean AI SDK result: no Inferada metadata extraction
const clean = await generateText({ model: inferada.passthrough('your-model-id'), prompt: 'Hello' })

Create a chat completion

{
  "model": "qwen3.5-35b",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Hello!" }
  ],
  "stream": false,
  "temperature": 0.7,
  "max_tokens": 1000
}

{
  "id": "chatcmpl-550e8400-e29b-41d4-a716-446655440000",
  "object": "chat.completion",
  "created": 1704900000,
  "model": "qwen3.5-35b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I help?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 8,
    "total_tokens": 33,
    "inferada": {
      "cost_eur": 0.000014
    }
  },
  "timings": {
    "total_ms": 870,
    "upstream_ms": 820,
    "tokens_per_second": 38.5
  },
  "inferada": {
    "request_id": "550e8400-e29b-41d4-a716-446655440000",
    "timings": {
      "total_ms": 870,
      "upstream_ms": 820,
      "tokens_per_second": 38.5
    }
  }
}

Streaming

With stream: true, the endpoint responds with Server-Sent Events and Content-Type: text/event-stream. Each chunk is an OpenAI-compatible delta frame, and [DONE] terminates the stream. The final success or error chunk includes top-level inferada metadata even when stream_options.include_usage is false. When usage is requested, a separate chunk with an empty choices array contains token counts and usage.inferada.

Streaming uses an idle timeout that resets whenever a chunk arrives. If the upstream produces no chunk within that window, the gateway aborts the stream with a service-failure error. Buffered requests use a fixed request deadline.

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"inferada":{"request_id":"...","timings":{"upstream_ms":820}}}

data: [DONE]

Some processors require the full response, so the gateway disables streaming for that request. See the Processors guide for details.

Processors

Processors add redaction, output correction, policy checks, or routing controls to a completion. See the Processors guide.

Tools

The gateway supports the OpenAI tools and tool_choice request fields. When you send tool definitions, the gateway forwards them to the model verbatim, streams tool_calls deltas back on the wire, and stays out of the execution loop: your client runs the tool and sends the result in a follow-up turn. When the request does not define tools, the model can use the catalogue tools attached by your organisation, which the gateway executes. See the Tools guide.

Timings

Every buffered completion includes the latency breakdown at inferada.timings. The same value is temporarily mirrored at top-level timings for existing integrations:

total_msnumber

End-to-end latency, including queue time and gateway processing.

upstream_msnumber

Time spent in the upstream model request.

tokens_per_secondnumber

Completion-token throughput computed from upstream_ms. Present only when completion tokens are greater than zero.

processors_msnumber

Measured aggregate of pre + post processor time. Only present when processors ran. See Processors.

queued_msnumber

Time spent waiting for a concurrency slot before processing began. Present only when greater than zero. A non-zero value indicates contention under a max_concurrent rule. Streaming responses expose the same value in the x-inferada-queued-ms header.

gateway_msnumber

Residual time spent in the gateway itself (validation, serialization, accounting writes). Only present when non-zero.

When the selected upstream is vLLM, its numeric performance metrics are preserved in the same object. This includes time_to_first_token_ms, generation_time_ms, and mean_itl_ms. To keep provider timings distinct from gateway timings, vLLM's queue_time_ms is exposed as upstream_queue_ms and its tokens_per_second as upstream_tokens_per_second. Standard OpenAI usage token counts and system_fingerprint remain in their standard top-level locations.

Model aliases

An organisation administrator can create a stable alias for one or more model IDs. For example, send model: "chat" instead of a version-specific model ID. The alias can later point to a different model without requiring a client change.

• Each alias maps to an ordered chain of real model_ids. The resolver selects the first active entry permitted by the token's allowed_models restriction.
• The gateway also tries the next eligible target when dispatch fails before a response is committed because of connectivity, timeout, throttling, upstream 5xx, or upstream 404. Caller-invalid and authentication responses such as 400, 401, 403, and 422 are terminal.
• Content processors run once and remain in force across attempts. A fallback target is eligible only when its effective processor policy matches the policy already executed; explicit request processor settings apply to every target. Route policies such as require_local are evaluated against every concrete candidate before any upstream request is sent.
• The standard response model remains the requested alias. Inspect inferada.routing for resolved_model, fallback_used, dispatch_attempts, and the optional fallback_reason.
• Once streaming output is visible, or a gateway-executed tool has produced effects, the route is fixed and no transparent retry occurs.
• Real model_ids always win over aliases: adding an alias never shadows an existing model.
• Aliases are discoverable via /v1/models: each model entry carries an aliases array listing every alias key that routes to it.
• When an administrator sunsets a model, its database identity and historical references are retained. Existing aliases promote their next active target; the old concrete model ID can also be retained as an alias to a chosen replacement.
• Aliases are scoped per service. The completions service's chat and the TTS service's chat would be independent.

List available models

# All models
curl https://api.inferada.com/v1/models -H "Authorization: Bearer inf_YOUR_TOKEN"

# Only embedding models
curl "https://api.inferada.com/v1/models?scope=embeddings" -H "Authorization: Bearer inf_YOUR_TOKEN"

{
  "object": "list",
  "data": [
    {
      "id": "qwen3.5-35b",
      "object": "model",
      "created": 1704900000,
      "owned_by": "inferada",
      "scope": "completions",
      "display_name": "Qwen 3.5 35B",
      "description": "General-purpose chat model with 128K context.",
      "aliases": ["chat", "smart"],
      "inferada": {
        "service_id": "01J6A...",
        "service_name": "LLM vLLM",
        "service_is_default": false,
        "processing": "local",
        "pricing": {
          "currency": "EUR",
          "input_per_mtok": 0.4,
          "cached_input_per_mtok": 0.1,
          "output_per_mtok": 1.2
        },
        "default_processors": ["anonymise"],
        "default_language": "nl",
        "tools": { "gateway": ["web_search"], "mcp_servers": [] },
        "processor_languages": { "anonymise": ["en", "nl"], "haystack": ["en", "nl"] }
      }
    }
  ]
}

Retrieve a model

{
  "id": "qwen3.5-35b",
  "object": "model",
  "created": 1704900000,
  "owned_by": "inferada",
  "scope": "completions",
  "display_name": "Qwen 3.5 35B",
  "description": "General-purpose chat model with 128K context.",
  "aliases": ["chat", "smart"],
  "inferada": {
    "service_id": "01J6A...",
    "processing": "local",
    "alias": {
      "name": "chat",
      "models": [
        { "id": "qwen3.5-35b", "processing": "local" },
        { "id": "qwen3.5-35b-cloud", "processing": "remote" }
      ]
    }
  }
}

Attach models in the portal

Models are configured per completion service. From Organisation → Models, an organisation administrator can use either workflow:

• Add model opens the complete form for one model. Select a template and upstream, enter model_id, and configure any overrides, tools, or limits.
• Add from templates attaches several global or organisation templates to one upstream. Each model uses the template name as its address. For a profile, that address is upstream_model_id:profile_name, e.g. gpt-4o:code), and inherited defaults resolve from the template at request time. The picker disables a template whose name is already attached to the selected service. Use Clone on the existing model to create a variant.

Both flows enforce one model_id per service, and a new model is callable as soon as the save completes.