InferadaDocsPortal
Guides

Rate limits

Configure usage and concurrency limits, then handle the 429 responses returned when a request exceeds them.

Limit hierarchy

Configure limits on the entity they should protect. The gateway evaluates each request at up to five levels:

  1. Service: limits attached to the service (e.g. the completions service).
  2. Model: per-model limits on a service model.
  3. Organisation: org-wide limits.
  4. User: limits on the authenticated user.
  5. Token: limits set directly on the API token.

If any level is exceeded, the gateway rejects the request with a 429. The response body tells you which level and which rule blocked it.

Limit definition

A periodic limit defines a metric, accounting period, and maximum. Organisation administrators configure these rules in the portal.

json
{
  "metric": "tokens",
  "period": "day",
  "max": 100000,
  "per_request": false
}
metricrequiredstring
tokens, prompt_tokens, completion_tokens, requests, audio_duration_seconds, or characters_synthesised.
periodrequiredstring
minute, day, week or month. Defines the accounting window.
maxrequirednumber
The maximum allowed value for this rule.
per_requestboolean
When true, the max applies to a single request (like a hard cap on max_tokens). When false (default), it aggregates over the period.

429: Limit exceeded

json
{
  "type": "limit_exceeded",
  "code": 429,
  "request_id": "...",
  "scope": "completions",
  "model_id": "qwen3.5-35b",
  "level": "model",
  "limit": {
    "metric": "tokens",
    "period": "day",
    "max": 100000
  },
  "current": 100000
}
typestring
Always "limit_exceeded" for this class of error.
levelstring
Which level breached: service, model, organisation, user or token.
limitobject
The exact rule that blocked the request.
currentnumber
Current counter value for a periodic limit.
requestednumber
Requested value for a per-request limit.

Retry after the reported accounting period resets. Avoid immediate retries against the same exhausted limit.

429: Concurrency limit

Concurrency limits are separate from periodic ones: they cap how many requests can be running at the same time. You can set them at any of the five levels. The gateway evaluates each level independently, and a request must acquire a slot at every applicable level. A request waits up to wait_timeout_ms; if no slot becomes available, it returns:

json
{
  "type": "concurrency_limit",
  "code": 429,
  "error": "Concurrency limit reached: 5 concurrent completions requests allowed at user level. Waited 30000ms.",
  "scope": "completions",
  "level": "user",
  "max_concurrent": 5,
  "waited_ms": 30000,
  "request_id": "..."
}

Define one with metric: "max_concurrent" and max: N (no period). Optional wait_timeout_ms controls how long a request queues before returning 429 (default 30000).

json
{
  "metric": "max_concurrent",
  "max": 5,
  "wait_timeout_ms": 30000
}

A concurrency response indicates temporary contention rather than an exhausted periodic quota. Retry with backoff.

When a request does acquire its slot, the gateway reports the wait time back so you can tell a slow response apart from a queued one:

  • Buffered completions: inferada.timings.queued_ms (only present when greater than 0). The legacy top-level timings alias temporarily mirrors it.
  • Streaming completions: x-inferada-queued-ms response header (always present, value in milliseconds). The same header also appears on buffered responses.

inferada.timings.upstream_ms remains the pure model-generation time and inferada.timings.processors_ms is measured (not a residual), so total_ms - upstream_ms - processors_ms - queued_ms is the gateway's own overhead, reported as inferada.timings.gateway_ms when non-zero.

Per-request vs period limits

A rule with per_request: true is a hard cap on a single call. For example a tokens per-request rule of 4096 effectively caps max_tokens to that value regardless of what the client sent. A rule with per_request: false is a running total that accumulates over the period and resets when the period rolls over.

See also