Create a batch inference job
Submit a batch from an uploaded input file (input_file_id) or inline request objects (requests); provide exactly one of the two. A batch has 24 hours from submission (expires_at) to finish: records not done by then come back with the batch_expired error and the batch ends expired. Standard batches bill at the discounted batch rate. Poll it with get, then download results via the output and error file ids. An API key whose spend budget is used up gets 402 key_budget_exceeded; a batch already running finishes, and its billed usage counts toward the key.
Required permission
Requires the batch.write permission. See Permissions for the roles and API key permissions that include it.
apiKeyAuthorizationBearer <token>OpenRelay API key. Send it as Authorization: Bearer or_… (legacy vl_… keys still work).
Idempotency-Key?stringMakes a retried create safe. Choose one value per create you mean to make, 1 to 512 bytes (bytes, not characters), such as a UUID, and send it again on every retry of that create. The first request with a key runs, and a 2xx answer is kept for 24 hours: a repeat with the same key and the same body gets that answer back, with the same status and the header Idempotent-Replayed: true, and creates nothing. A batch create compares the body byte for byte; a VM or pod create compares the fields it decodes, so whitespace and key order do not count. A repeat while the first request is still running answers 409; repeat it a second later. The same key with a different body answers 422. An answer other than a 2xx is not kept, so a retry after a 4xx or a 5xx runs again. A 2xx too large to keep holds the key without it: every repeat answers 409 and creates nothing. Keys are scoped to the organization and the operation.
1 <= lengthapplication/json- body
input_file_id?stringId of an uploaded JSONL input file (from the file upload).
requests?array<>Inline request objects (max 50,000), each with a custom_id and the request body for the chosen endpoint. Alternative to input_file_id for small batches.
endpoint*stringThe API family every record in the batch calls.
"/v1/chat/completions""/v1/completions""/v1/embeddings"completion_window*"24h"Time budget for the batch; only "24h" is supported.
"24h"model?stringOptional display hint shown on the batch immediately; validation settles the authoritative value from the input file.
priority?stringScheduling tier. standard (the default) runs at the discounted batch rate. expedited is granted a freeing pool slot ahead of any waiting standard batch, grows the fleet to a tighter drain target, and is billed at the online rate with no batch discount; it is available to every organization unless an operator has disabled it for yours, in which case the request is rejected with 400. Neither tier changes the 24-hour expiry.
"standard""standard""expedited"metadata?Key-value pairs echoed back on the batch object. Updating metadata later allows at most 32 pairs.
OK
application/json- response
A batch inference job (OpenAI-compatible).
id*stringBatch id (batch_…).
object*"batch""batch"endpoint*stringThe API family every record in the batch calls.
input_file_id*stringcompletion_window*"24h""24h"priority*stringThe scheduling tier this batch runs and is billed at. Batches created before the tier existed read as standard.
"standard""expedited"status*stringLifecycle state. validating → in_progress → finalizing → completed | failed | expired; cancelling → cancelled. A batch is in_progress while it waits for GPU capacity too; in_progress_at says when its requests started running.
"validating""in_progress""finalizing""completed""failed""expired""cancelling""cancelled"model?stringModel id, settled from the input file during validation.
output_file_id?stringPresent once results are written; fetch its content for successful records.
error_file_id?stringPresent once results are written; fetch its content for failed records.
created_at*integerUnix timestamp (seconds).
int64in_progress_at*integerUnix timestamp (seconds) when the batch's first requests started running. Always present and null until then, so an in_progress batch with a null in_progress_at is still queued for GPU capacity.
int64expires_at?integerWhen the completion window closes (created_at + 24h).
int64completed_at?integerint64failed_at?integerint64expired_at?integerint64cancelled_at?integerint64request_counts*Progress counters, settled as the batch validates and shards complete.
usage?Rolled-up token and cost totals, present once any progress is recorded. cost_nano_usd is the price you pay (batch discount applied), in nano-USD so sub-cent batches stay exact.
metadata?Your key-value pairs, echoed back unchanged.
curl -X POST 'https://inference.openrelay.inc/v1/batches' \ -H 'Content-Type: application/json' \ -d '{ "endpoint": "/v1/chat/completions", "completion_window": "24h"}'{ "id": "string", "object": "batch", "endpoint": "string", "input_file_id": "string", "completion_window": "24h", "priority": "standard", "status": "validating", "model": "string", "output_file_id": "string", "error_file_id": "string", "created_at": 0, "in_progress_at": 0, "expires_at": 0, "completed_at": 0, "failed_at": 0, "expired_at": 0, "cancelled_at": 0, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "usage": { "input_tokens": 0, "output_tokens": 0, "cost_nano_usd": 0 }, "metadata": { "property1": "string", "property2": "string" }}List batches, newest first GET
Returns one page of your organization's batches, newest first, in the OpenAI list envelope. Page with limit (1 to 100, default 20) and after.
Get a batch by id GET
Returns one batch with its status, priority and request counts, plus its usage once any is recorded and the output and error file ids once written. Poll it to track progress.