Overview

The Batch API runs large jobs of inference requests asynchronously. You upload a JSONL file of requests (or send them inline), submit a batch, and the gateway works through it within a 24-hour window. When it finishes, you download a JSONL file of results. Batch requests are billed at a lower per-token rate than online requests.

It is OpenAI-compatible: the Files and Batches endpoints, the request and response shapes, the status lifecycle, and the JSONL record formats match the OpenAI Batch API, so existing OpenAI batch tooling works against it.

Base URL

OpenRelay has two API hosts, and batch lives on exactly one of them:

Every batch and file endpoint below is on inference.openrelay.inc, versioned under /v1. Sending a batch request to api.openrelay.inc returns 404; the control plane does not serve these routes. Both hosts accept the same vl_ API key and Authorization: Bearer header. See Authentication.

Three clients cover most workflows, all shown in the Quickstart:

• curl (or any HTTP client) against the endpoints below.
• The orl CLI: orl batches create --input-file requests.jsonl --endpoint /v1/chat/completions uploads and submits in one command, then orl batches get, orl batches list, orl batches cancel, and orl files content get <id> -o results.jsonl cover the rest of the lifecycle.
• The OpenAI SDKs, pointed at this base URL: client.files.create, client.batches.create, and friends work unchanged.

How it works

1. Prepare a JSONL file where each line is one request, keyed by a custom_id you choose. See Input format.
2. Upload the file with the Files API, which returns a file_id. For small jobs you can skip this and send the requests inline.
3. Create a batch from that file (or those inline requests) against an endpoint like /v1/chat/completions. See Batches.
4. Poll the batch until it reaches a terminal status. It moves through validating → in_progress → finalizing → completed.
5. Download the output_file_id (successful requests) and, if present, the error_file_id (failed requests). See Results.

Limits

A batch that is not finished within its 24-hour window expires: completed work is kept and billed, and unfinished requests are written to the error file with code: "batch_expired".

Best practices

Make batches big. Every batch has a fixed startup period before the first result, so throughput and time-to-results are best when you pack one batch with as many requests as fit (up to 50,000 requests / 200 MB) instead of submitting many small ones. Do not create one batch per document: put every page of every document in one file and encode the document in your custom_id (inv-0042-page-003), then join results back per document when you download.

Compress images before you encode them. For image inputs (OCR, vision), base64 inflates bytes by about a third, and resolution beyond roughly 1,600 px on the long side does not improve results. A page scanned as raw PNG can run 2 MB or more; the same page as a JPEG is typically 200 to 400 KB, so even a 90-page document fits in a file with room for hundreds more. If a large upload returns 413, image compression is almost always the fix.

One model per batch. A batch runs a single model; records naming a different model fail individually with model_mismatch. Split multi-model workloads into one batch per model.

Endpoints

Input format

Each line of the input file is one request object, matching the OpenAI batch input shape:

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gpt-oss-120b", "messages": [{"role": "user", "content": "Name three primary colors."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gpt-oss-120b", "messages": [{"role": "user", "content": "Name three prime numbers."}]}}

Pricing

Batch requests are metered per token exactly like online requests, but at a reduced rate (roughly half the online rate). A batch's rolled-up token counts and cost appear in the usage object on the batch as it runs. See Pricing & billing.

Guides