Overview

The Batch API runs large jobs of inference requests asynchronously. You upload a JSONL file of requests (or send them inline), submit a batch, and the gateway works through it within a 24-hour window. When it finishes, you download a JSONL file of results. Batch requests are billed at a lower per-token rate than online requests.

It is OpenAI-compatible: the Files and Batches endpoints, the request and response shapes, the status lifecycle, and the JSONL record formats match the OpenAI Batch API, so existing OpenAI batch tooling works against it.

Base URL

OpenRelay has two API hosts, and batch lives on exactly one of them:

Every batch and file endpoint below is on inference.openrelay.inc, versioned under /v1. Sending a batch request to api.openrelay.inc returns 404; the control plane does not serve these routes. Both hosts accept the same or_ API key and Authorization: Bearer header. See Authentication.

Three clients cover most workflows, all shown in the Quickstart:

• curl (or any HTTP client) against the endpoints below.
• The orl CLI: orl batches create --input-file requests.jsonl --endpoint /v1/chat/completions uploads and submits in one command, then orl batches get, orl batches list, orl batches cancel, and orl files content get <id> -o results.jsonl cover the rest of the lifecycle.
• The OpenAI SDKs, pointed at this base URL: client.files.create, client.batches.create, and friends work unchanged.

How it works

1. Prepare a JSONL file where each line is one request, keyed by a custom_id you choose. See Input format.
2. Upload the file with the Files API, which returns a file_id. For small jobs you can skip this and send the requests inline.
3. Create a batch from that file (or those inline requests) against an endpoint like /v1/chat/completions. See Batches.
4. Poll the batch until it reaches a terminal status. It moves through validating → in_progress → finalizing → completed.
5. Download the output_file_id (successful requests) and, if present, the error_file_id (failed requests). See Results.

Limits

A batch that is not finished within its 24-hour window expires: completed work is kept and billed, and unfinished requests are written to the error file with code: "batch_expired".

Best practices

Make batches big. Every batch has a fixed startup period before the first result, so throughput and time-to-results are best when you pack one batch with as many requests as fit (up to 50,000 requests / 200 MB) instead of submitting many small ones. Do not create one batch per document: put every page of every document in one file and encode the document in your custom_id (inv-0042-page-003), then join results back per document when you download.

Compress images before you encode them. For image inputs (OCR, vision), base64 inflates bytes by about a third, and resolution beyond roughly 1,600 px on the long side does not improve results. A page scanned as raw PNG can run 2 MB or more; the same page as a JPEG is typically 200 to 400 KB, so even a 90-page document fits in a file with room for hundreds more. If a large upload returns 413, image compression is almost always the fix.

One model per batch. A batch runs a single model; records naming a different model fail individually with model_mismatch. Split multi-model workloads into one batch per model.

Size a tool-calling batch by its rounds, not its records. A record with openrelay.tool_config is re-invoked after every tool round, so it costs up to max_rounds + 1 model invocations rather than one. A full 50,000-record batch at the max_rounds cap of 16 is up to 850,000 model invocations, and each round also re-sends the growing message history as the prompt. Start with the default max_rounds of 8, set it explicitly to the smallest number your task actually needs, and run a small batch first to see the real average round count before scaling up.

Endpoints

Input format

Each line of the input file is one request object, matching the OpenAI batch input shape:

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gpt-oss-120b", "messages": [{"role": "user", "content": "Name three primary colors."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gpt-oss-120b", "messages": [{"role": "user", "content": "Name three prime numbers."}]}}

Tool calling in a batch (openrelay.tool_config)

A /v1/chat/completions record may carry standard OpenAI tools / tool_choice fields plus one OpenRelay extension, tool_config, that tells the batch worker how to execute the model's tool calls between turns. It lives under the reserved openrelay object on the body, the same namespace the response side uses for tool_rounds:

"openrelay": {
  "tool_config": {
    "url": "https://your-app.example.com/api/openrelay-tools",
    "authorization": "Bearer <token>",
    "context": { "contractId": "..." },
    "max_rounds": 8,
    "timeout_ms": 30000
  }
}

// before (no longer accepted)
"body": { "model": "...", "messages": [], "tool_config": { "url": "..." } }
// after
"body": { "model": "...", "messages": [], "openrelay": { "tool_config": { "url": "..." } } }

Omitted or 0 values for max_rounds and timeout_ms take the defaults, and values above the caps are clamped down to them; negative values are rejected at validation.

When a turn finishes with finish_reason: "tool_calls", the worker POSTs { "custom_id", "context", "tool_calls" } to url and expects { "results": [{ "tool_call_id", "content" }] } back. It appends the assistant message and one role: "tool" message per result, then re-invokes the model, up to max_rounds times.

• openrelay.tool_config is stripped before every model invocation and never appears in output or error files. If openrelay carries nothing else, the whole object is removed too, so the model sees exactly the body you would have written by hand.
• The final output line's usage sums prompt and completion tokens across all turns of the record (that is also what you are billed on), and the body carries "openrelay": { "tool_rounds": N } when any round ran.
• An executor failure (non-2xx, timeout, or a malformed response) fails that record with code tool_execution_failed; a malformed tool_config fails it at validation with invalid_tool_config, and the retired bare spelling fails it with tool_config_moved.
• If max_rounds is exhausted, the last model response is stored as a normal output line with finish_reason: "tool_calls", matching OpenAI.
• Records without openrelay.tool_config behave exactly as before, even when they carry tools.
• Make your executor idempotent. Batch processing retries and resumes internally, so a record's tool calls can be executed more than once (the same tool_call_id may hit your endpoint again). Executors must be idempotent or tolerate duplicate calls, and a retried multi-round record's stored usage reflects the run that produced its final output.
• A record that fails mid-loop (for example an executor error on round 3) produces an error line, and the tokens its earlier turns consumed are not billed.

MCP executors ("type": "mcp")

Set type to "mcp" to back a record's tools with any MCP server speaking Streamable HTTP (JSON-RPC 2.0 over a single POST endpoint, protocol revision 2025-03-26 or later), instead of implementing the plain POST protocol above:

"openrelay": {
  "tool_config": {
    "type": "mcp",
    "url": "https://your-mcp-server.example.com/mcp",
    "authorization": "Bearer <token>",
    "context": { "contractId": "..." }
  }
}

• Once per record loop (cached across rounds) the worker sends initialize (with clientInfo.name openrelay-batch-worker) followed by notifications/initialized, honoring any Mcp-Session-Id the server issues. Stateless servers that answer initialize with an HTTP 404 or 405 (no handshake endpoint) are handled: the worker falls back to sending tools/call directly. Any other failing status, including 401 and 403, fails the record rather than silently proceeding unauthenticated.
• Each model tool call becomes one tools/call request whose name is the function name and whose arguments is the parsed JSON object from the model's arguments string. Responses may be application/json or text/event-stream (the final JSON-RPC message of the stream is used).
• The tool result delivered back to the model is the concatenation of the result's content items of type "text". A result with isError: true is not a record failure: its text is delivered to the model as the tool content, matching MCP semantics, so the model can react to it.
• MCP tools/call has no side channel for context, so when openrelay.tool_config.context is present it is merged into every call's arguments under the reserved _context key (omitted entirely when context is absent). Don't define a tool parameter named _context.
• Transport errors, non-2xx responses, and JSON-RPC error responses fail the record with tool_execution_failed, exactly like the http type.
• Everything else is unchanged from the http type: authorization is sent on every request, timeout_ms bounds each request, max_rounds bounds the loop, and the same idempotency advice applies.

Pricing

Batch requests are metered per token exactly like online requests, but at a reduced rate (roughly half the online rate). A batch's rolled-up token counts and cost appear in the usage object on the batch as it runs. See Pricing & billing.

Guides