Batches

A batch runs a set of inference requests asynchronously against one endpoint. You create it from an uploaded file or from inline requests, poll it to a terminal status, then download its result files.

The batch objectCopy Anchor Link

Create a batchCopy Anchor Link

POST https://inference.openrelay.inc/v1/batches

Every record is checked when you create the batch: records that break a rule are rejected and the rest are accepted. The create returns the batch at status validating, with rejected_count, the number of records it rejected, and request_counts already set: total is every record in the input, and the rejected records count as failed.

A key with a spend budget that is used up gets 402 key_budget_exceeded and creates nothing. The check is only whether the budget is used up, so one batch can pass what is left of it; a batch already running finishes, and its billed usage counts toward the key.

curl https://inference.openrelay.inc/v1/batches \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-a1b2c3d4e5f6a7b8c9d0e1f2",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h",
    "metadata": { "job": "nightly-summaries" }
  }'

{
  "id": "batch_9f8e7d6c5b4a",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "input_file_id": "file-a1b2c3d4e5f6a7b8c9d0e1f2",
  "completion_window": "24h",
  "priority": "standard",
  "status": "validating",
  "model": "openrelay/gemma-4-31b",
  "created_at": 1752460801,
  "in_progress_at": null,
  "expires_at": 1752547201,
  "request_counts": { "total": 2, "completed": 0, "failed": 0 },
  "metadata": { "job": "nightly-summaries" }
}

CLI: orl batches create --input-file-id file-a1b2c3d4e5f6a7b8c9d0e1f2 --endpoint /v1/chat/completions (or --input-file requests.jsonl to upload and create in one command; completion-window defaults to 24h).

Rejected recordsCopy Anchor Link

The create checks every record before anything runs: its shape, its openrelay extensions, and its model. A record that breaks a rule is rejected, never runs and is never billed; the rest of the batch runs. The create's 200 carries fields that appear only on the create answer: rejected_count, and when the create rejects records, the other two:

line is the record's line in the input file, counting from 1 and counting blank lines, so it is the number your editor shows; for inline requests it is the request's position in requests, from 1. custom_id is empty when the line could not be read as a record (invalid_json) or names none. One whose JSON-escaped form is longer than 512 bytes is cut short and ends in ....

{
  "id": "batch_9f8e7d6c5b4a",
  "object": "batch",
  "status": "validating",
  "model": "openrelay/gemma-4-31b",
  "request_counts": { "total": 4, "completed": 0, "failed": 3 },
  "rejected_count": 3,
  "rejected_records": [
    { "line": 1, "custom_id": "req-1", "code": "duplicate_custom_id", "message": "custom_id appears more than once" },
    { "line": 2, "custom_id": "req-2", "code": "streaming_unsupported", "message": "\"stream\": true is not supported in batch" },
    { "line": 4, "custom_id": "req-1", "code": "duplicate_custom_id", "message": "custom_id appears more than once" }
  ],
  "rejected_records_file_id": "file-0f1e2d3c4b5a69788796a5b4",
  "…": "…"
}

The batch's error file lists the rejected records too, once the batch finishes.

When no record can run, the create returns 400 no_valid_records, creates no batch, and carries the same fields under error, with the request_counts the batch would have had:

{
  "error": {
    "message": "all 2 records were rejected; see rejected_records",
    "type": "invalid_request_error",
    "code": "no_valid_records",
    "rejected_records": [
      { "line": 1, "custom_id": "req-1", "code": "model_not_offered", "message": "model \"openrelay/unknown\" is not offered for batch (offered: openrelay/bge-m3, openrelay/deepseek-ocr-2, openrelay/gemma-4-31b)" },
      { "line": 2, "custom_id": "", "code": "invalid_json", "message": "invalid character 'o' in literal null (expecting 'u')" }
    ],
    "rejected_records_file_id": "file-6a5b4c3d2e1f0a9b8c7d6e5f",
    "rejected_count": 2,
    "request_counts": { "total": 2, "completed": 0, "failed": 2 }
  }
}

A record is rejected with one of these codes:

Some answers refuse the whole create instead, and create nothing:

A 409 feature_disabled comes before any record is checked. "priority": "expedited" is refused before the rest of the request is checked or the input is read. A record carrying openrelay.workflow anywhere in the input refuses the create whatever the other records hold, even past the 50,000-record limit or when the input cannot be read to its end after that record, so the input gets no input_too_large, read error, no_valid_records or rejected records instead.

CLI: orl batches create prints the batch on stdout and, when the create rejected records, a preview of up to 10 of them and the rejections file id on stderr (exit 0); --rejections-out rejections.jsonl writes every rejection to a file. When no record can run the error carries the preview (exit 2), and a 409 feature_disabled exits 6.

Inline requestsCopy Anchor Link

For small jobs, skip the upload and pass requests inline. Each element is one request object, the same shape as an input file line:

curl https://inference.openrelay.inc/v1/batches \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h",
    "requests": [
      {"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gemma-4-31b", "messages": [{"role": "user", "content": "Name three primary colors."}]}}
    ]
  }'

Retry a create safelyCopy Anchor Link

A create that timed out or lost its connection may still have made the batch, and a plain retry would run and bill the work twice. Send an Idempotency-Key header with a value you choose once per batch (1 to 512 bytes, such as a UUID), and send the same key with the same body on every retry:

curl https://inference.openrelay.inc/v1/batches \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: nightly-summaries-2026-10-15" \
  -d '{
    "input_file_id": "file-a1b2c3d4e5f6a7b8c9d0e1f2",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

The 200 is kept for 24 hours. Keys belong to your organization, and the body must be byte-identical, so keep the bytes you sent the first time. A retry is checked like any create first: it needs batch.write and passes the balance check. The full rules are in Idempotent creates.

File uploads take no key: a repeated upload stores a second file, with a new id, which costs nothing until a batch runs it. To retry a create after an upload, reuse the first file's input_file_id: a new id is a different body under the same key, and gets a 422.

CLI: orl batches create --idempotency-key nightly-summaries-2026-10-15 sends the key. It also tags the batch with it, so a retry of --input-file finds the first batch before uploading the file again.

PriorityCopy Anchor Link

A batch runs at one of two tiers, chosen at create and fixed for its life:

Expedited is a queue preference, not a deadline: there is no SLA, no credit if it is slow, and a running batch is never interrupted to make room for it. Everything else about the batch surface (file input, JSONL results, the 50,000-record cap, the 24-hour window) is identical between tiers.

Expedited is available to every organization. An operator can turn it off for a single organization; if yours is turned off, a create with "priority": "expedited" returns 409 feature_disabled with the message priority "expedited" is not enabled for this organization and creates nothing, rather than quietly running at standard.

curl https://inference.openrelay.inc/v1/batches \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-a1b2c3d4e5f6a7b8c9d0e1f2",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h",
    "priority": "expedited"
  }'

priority is an OpenRelay field, so the OpenAI SDKs send it through extra_body:

batch = client.batches.create(
    input_file_id="file-a1b2c3d4e5f6a7b8c9d0e1f2",
    endpoint="/v1/chat/completions",
    completion_window="24h",
    extra_body={"priority": "expedited"},
)

Read the tier back off the priority field on the response rather than assuming your request was honored. On the CLI it is orl batches create --priority expedited.

Get a batchCopy Anchor Link

GET https://inference.openrelay.inc/v1/batches/{id}

curl https://inference.openrelay.inc/v1/batches/batch_9f8e7d6c5b4a \
  -H "Authorization: Bearer $OPENRELAY_API_KEY"

Returns the batch object, or 404 if the id is unknown to your organization. Poll this until status is terminal. CLI: orl batches get batch_9f8e7d6c5b4a.

List batchesCopy Anchor Link

GET https://inference.openrelay.inc/v1/batches?limit=20&after=batch_9f8e7d6c5b4a

Returns your batches newest first.

{
  "object": "list",
  "data": [ { "id": "batch_9f8e7d6c5b4a", "object": "batch", "status": "completed", "…": "…" } ],
  "first_id": "batch_9f8e7d6c5b4a",
  "last_id": "batch_9f8e7d6c5b4a",
  "has_more": false
}

To page, pass the previous response's last_id as after; stop when has_more is false. CLI: orl batches list --limit 20 --after batch_9f8e7d6c5b4a.

Cancel a batchCopy Anchor Link

POST https://inference.openrelay.inc/v1/batches/{id}/cancel

Cancels a batch that is still running. In-flight requests finish, unstarted requests are not started, partial results are written, and the batch settles to cancelled within a few minutes. The response echoes the batch with status cancelling.

curl https://inference.openrelay.inc/v1/batches/batch_9f8e7d6c5b4a/cancel \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -X POST

Cancelling a batch that has already reached a terminal status returns 409. CLI: orl batches cancel batch_9f8e7d6c5b4a.

Update metadataCopy Anchor Link

POST https://inference.openrelay.inc/v1/batches/{id}/metadata

Replaces the batch's metadata map (labels, tags, notes). Unlike cancel, this is editable at any time, including after the batch is terminal: metadata is your own data, not lifecycle state, so you can re-tag and organize finished jobs. The whole desired map is sent (send the full set you want, not a delta); an empty object clears all metadata. Keys are capped at 64 characters and values at 512, with at most 32 keys.

The dashboard uses a reserved tags key (a comma-separated list) to render color-coded tag chips and drive the filters on the Batch page, so tags you set over the API show up there too.

curl https://inference.openrelay.inc/v1/batches/batch_9f8e7d6c5b4a/metadata \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "metadata": { "tags": "nightly-eval,customer-42", "job": "nightly-summaries" } }'

The response echoes the full batch object with the updated metadata. A batch owned by another organization (or an unknown id) returns 404.

Status lifecycleCopy Anchor Link

A batch moves through these statuses, matching the OpenAI Batch API exactly:

completed, failed, expired, and cancelled are terminal.

Queued or runningCopy Anchor Link

A batch reads in_progress from the moment validation passes, including while it waits for a free GPU slot on its model, so OpenAI SDK clients see only the statuses they expect. in_progress_at is what separates the wait from the run: it is null while the batch is queued and becomes the time its first requests started once a slot frees.

{ "status": "in_progress", "created_at": 1759395600, "in_progress_at": null }

The gap between created_at and in_progress_at is how long the batch queued, which is the wait expedited priority shortens. Nothing is charged while a batch waits, and the 24-hour window runs from created_at, so time spent queued counts against it.