Skip to content
OpenRelay is in early access, and the /v1 API is stable. New capabilities ship in the changelog.
Skip to Content
GuidesInference APIsAsync API

Limits and errors

The size limits of an async job, the errors a create can answer with, and the codes a failed request carries in its result.

Async is in beta

Shapes and limits can still change before general availability.

Limits

1 hour24 hours
Requests per job10,00050,000
Inline requests per create1,0001,000
Input file200 MB200 MB
One model call30 minutes30 minutes
Retention of a job and its files30 days30 days

A request answer over 64 MB fails that request with response_too_large. Your key's budget and balance apply as for any inference. The routes need Batch: Read (list, get, results) and Batch: Write (create, cancel) permissions.

There is no capacity refusal: every job is accepted, and a request that does not start before its deadline expires at no charge.

Errors at create

The answer is the error shape with these codes:

StatusCodeWhen
400invalid_request_errorA bad completion_window, both or neither of input_file_id and requests, or more than 1,000 inline requests.
400input_too_largeMore lines than the window allows.
400no_valid_recordsNo line can run. A line is judged on its own, and the create fails only when every line is rejected.
400input_unreadable, input_file_changedThe input file's content is missing or is not what its upload recorded. Upload it again.
402insufficient_balance, key_budget_exceededAs for Sync inference.
403The key may not use Async, or a hold applies, as for Sync inference.
404Async is not turned on for the organization.
409idempotency_key_in_useA create with the same Idempotency-Key is still running.
422idempotency_key_reusedThe key was used with a different body.
503catalog_unavailable, orchestrator_unavailableRetry.

Why a line is rejected

A rejected line is counted in rejected_count and listed with a code:

CodeMeaning
streaming_unsupportedThe body asks to stream.
invalid_urlThe url is not /v1/chat/completions, /v1/messages or /v1/embeddings.
model_not_offeredThe model is not offered for Async in this window. The message names the window and lists the models offered.
model_not_allowedThe key may not use the model.
invalid_bodyThe body is not a JSON object, repeats a key the create reads (model, stream, tools, tool_choice, openrelay), or carries an openrelay extension an async job does not run.
invalid_requestThe line failed validation.

Why a request does not complete

A result with status other than completed carries an error.code:

CodeStatusMeaningCharged
deadline_expiredexpiredIt never started before deadline_at.No
cancelledcancelledThe job was cancelled before it started.No
model_errorfailedThe model answered non-2xx. error.status_code and error.body carry its answer.No
model_unavailablefailedThe model was withdrawn or disabled for Async while the request waited.No
request_failedfailedThe request was sent and may have run, then could not finish.No
request_time_limitfailedThe model call hit the 30 minute limit.No
response_too_largefailedThe answer was over the size limit.No

Only a completed request is charged. See Pricing.

On this page