OpenRelay API Documentation

Create, get, list, and cancel batches. The batch object, inline vs file input, the status lifecycle, and pagination.

A batch runs a set of inference requests asynchronously against one endpoint. You create it from an uploaded file or from inline requests, poll it to a terminal status, then download its result files.

The batch object

Field	Type	Description
`id`	string	The batch id (`batch_…`).
`object`	string	Always `"batch"`.
`endpoint`	string	The endpoint the requests target, e.g. `/v1/chat/completions`.
`input_file_id`	string	The input file the batch runs. For inline requests, an input file the gateway synthesized from them.
`completion_window`	string	Always `"24h"`.
`status`	string	The lifecycle status. See Status lifecycle.
`model`	string	The model id, once known.
`output_file_id`	string	Present when there are successful results (the file of response lines).
`error_file_id`	string	Present when there are failed results (the file of error lines).
`created_at`	integer	Unix seconds when the batch was submitted.
`expires_at`	integer	Unix seconds when the 24-hour window ends.
`completed_at` / `failed_at` / `expired_at` / `cancelled_at`	integer	Unix seconds the batch reached that terminal status (whichever applies).
`request_counts`	object	`{ total, completed, failed }`, updated as work progresses.
`usage`	object	Rolled-up `{ input_tokens, output_tokens, cost_nano_usd }`. Present once non-zero. `cost_nano_usd` is the customer price in nanodollars (1e-9 USD), with the batch discount already applied.
`metadata`	object	The string-to-string map you attached at creation.

Create a batch

POST https://inference.openrelay.inc/v1/batches

Field	Type	Description
`input_file_id`	string	The id of an uploaded `purpose: "batch"` file. Provide either this or `requests`, not both.
`requests`	array	Inline request objects (the same shape as input file lines). The gateway stores them as a synthesized input file. Provide either this or `input_file_id`.
`endpoint`	string (required)	One of `/v1/chat/completions` or `/v1/completions`.
`completion_window`	string (required)	Must be `"24h"`.
`model`	string	Optional hint so the batch shows a model id immediately. The authoritative value is read from the input file when the batch validates.
`metadata`	object	Optional string-to-string map, echoed back on the batch.

Creating a batch returns it at status validating with zeroed request_counts; the real counts settle once the input file is validated.

curl https://inference.openrelay.inc/v1/batches \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-a1b2c3d4e5f6a7b8c9d0e1f2",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h",
    "metadata": { "job": "nightly-summaries" }
  }'

{
  "id": "batch_9f8e7d6c5b4a",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "input_file_id": "file-a1b2c3d4e5f6a7b8c9d0e1f2",
  "completion_window": "24h",
  "status": "validating",
  "created_at": 1752460801,
  "expires_at": 1752547201,
  "request_counts": { "total": 0, "completed": 0, "failed": 0 },
  "metadata": { "job": "nightly-summaries" }
}

CLI: orl batches create --input-file-id file-a1b2c3d4e5f6a7b8c9d0e1f2 --endpoint /v1/chat/completions (or --input-file requests.jsonl to upload and create in one command; completion-window defaults to 24h).

Inline requests

For small jobs, skip the upload and pass requests inline. Each element is one request object, the same shape as an input file line:

curl https://inference.openrelay.inc/v1/batches \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h",
    "requests": [
      {"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gpt-oss-120b", "messages": [{"role": "user", "content": "Name three primary colors."}]}}
    ]
  }'

Provide exactly one of input_file_id or requests: sending both, or neither, returns 400. Inline requests are capped at 50,000 records like an uploaded file.

Get a batch

GET https://inference.openrelay.inc/v1/batches/{id}

curl https://inference.openrelay.inc/v1/batches/batch_9f8e7d6c5b4a \
  -H "Authorization: Bearer $OPENRELAY_API_KEY"

Returns the batch object, or 404 if the id is unknown to your organization. Poll this until status is terminal. CLI: orl batches get batch_9f8e7d6c5b4a.

List batches

GET https://inference.openrelay.inc/v1/batches?limit=20&after=batch_9f8e7d6c5b4a

Returns your batches newest first.

Query	Description
`limit`	Page size, 1 to 100. Defaults to 20.
`after`	A batch id to page after (the `last_id` of the previous page).

{
  "object": "list",
  "data": [ { "id": "batch_9f8e7d6c5b4a", "object": "batch", "status": "completed", "…": "…" } ],
  "first_id": "batch_9f8e7d6c5b4a",
  "last_id": "batch_9f8e7d6c5b4a",
  "has_more": false
}

To page, pass the previous response's last_id as after; stop when has_more is false. CLI: orl batches list --limit 20 --after batch_9f8e7d6c5b4a.

Cancel a batch

POST https://inference.openrelay.inc/v1/batches/{id}/cancel

Cancels a batch that is still running. In-flight requests finish, unstarted requests are not started, partial results are written, and the batch settles to cancelled within a few minutes. The response echoes the batch with status cancelling.

curl https://inference.openrelay.inc/v1/batches/batch_9f8e7d6c5b4a/cancel \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -X POST

Cancelling a batch that has already reached a terminal status returns 409. CLI: orl batches cancel batch_9f8e7d6c5b4a.

Status lifecycle

A batch moves through these statuses, matching the OpenAI Batch API exactly:

Status	Meaning
`validating`	The input file is being checked before the run starts.
`in_progress`	Requests are being run.
`finalizing`	The run finished; the result files are being assembled.
`completed`	Done. `output_file_id` (and `error_file_id`, if any) are ready.
`failed`	The batch could not run, for example an unreadable or over-limit input file.
`expired`	The 24-hour window elapsed before the batch finished. Completed work is kept and billed; unfinished requests are written to the error file with `code: "batch_expired"`.
`cancelling`	A cancel was requested; the batch is winding down.
`cancelled`	The batch was cancelled.

completed, failed, expired, and cancelled are terminal.

Batches