Overview

The Batch API runs large jobs of inference requests asynchronously. You upload a JSONL file of requests (or send them inline), submit a batch, and the gateway works through it within a 24-hour window. When it finishes, you download a JSONL file of results. At the default standard priority, batch requests are billed at 50% of the Sync rate, the rate for real-time requests.

It is OpenAI-compatible: the Files and Batches endpoints, the request and response shapes, the status lifecycle, and the JSONL record formats match the OpenAI Batch API, so existing OpenAI batch tooling works against it.

Base URLCopy Anchor Link

OpenRelay has two API hosts, and batch lives on exactly one of them:

Every batch and file endpoint below is on inference.openrelay.inc, versioned under /v1. Sending a batch request to api.openrelay.inc returns 404; the control plane does not serve these routes. Both hosts accept the same or_ API key and Authorization: Bearer header. See Authentication.

Three clients cover most workflows, all shown in the Quickstart:

• curl (or any HTTP client) against the endpoints below.
• The orl CLI: orl batches create --input-file requests.jsonl --endpoint /v1/chat/completions uploads and submits in one command, then orl batches get, orl batches list, orl batches cancel, and orl files content get <id> -o results.jsonl cover the rest of the lifecycle.
• The OpenAI SDKs, pointed at this base URL: client.files.create, client.batches.create, and friends work unchanged.

How it worksCopy Anchor Link

1. Prepare a JSONL file where each line is one request, keyed by a custom_id you choose. See Input format.
2. Upload the file with the Files API, which returns a file_id. For small jobs you can skip this and send the requests inline.
3. Create a batch from that file (or those inline requests) against an endpoint like /v1/chat/completions. Every record is checked then: the answer lists the records it rejected, and the rest run. See Batches.
4. Poll the batch until it reaches a terminal status. It moves through validating → in_progress → finalizing → completed.
5. Download the output_file_id (successful requests) and, if present, the error_file_id (failed requests). See Results.

LimitsCopy Anchor Link

A batch that is not finished within its 24-hour window expires: completed work is kept and billed, and unfinished requests are written to the error file with code: "batch_expired".

Each model call in a batch has 30 minutes to return its response, counted from when the call is sent to the model, so time it waits for the model to start on it counts. A call that has not returned its response by then is stopped, is not retried, and is not billed. Its record gets an error line with code: "request_time_limit". The one exception is a call whose model server failed under it: that record runs again on new capacity. Each turn of a tool-calling record is its own call, and a turn stopped at the limit fails the record. A request a workflow makes is a call too: one stopped at the limit comes back to your callback as a failed request, and the record keeps running.

Best practicesCopy Anchor Link

Make batches big. Every batch has a fixed startup period before the first result, so throughput and time-to-results are best when you pack one batch with as many requests as fit (up to 50,000 requests / 200 MB) instead of submitting many small ones. Do not create one batch per document: put every page of every document in one file and encode the document in your custom_id (inv-0042-page-003), then join results back per document when you download.

Compress images before you encode them. For image inputs (OCR, vision), base64 inflates bytes by about a third, and resolution beyond roughly 1,600 px on the long side does not improve results. A page scanned as raw PNG can run 2 MB or more; the same page as a JPEG is typically 200 to 400 KB, so even a 90-page document fits in a file with room for hundreds more. If a large upload returns 413, image compression is almost always the fix.

One model per batch. A batch runs a single model; records naming a different model are rejected individually at create with model_mismatch. Split multi-model workloads into one batch per model.

Size a tool-calling batch by its rounds, not its records. A record with openrelay.tool_config is re-invoked after every tool round, so it costs up to max_rounds + 1 model invocations rather than one. A full 50,000-record batch at the max_rounds cap of 16 is up to 850,000 model invocations, and each round also re-sends the growing message history as the prompt. Start with the default max_rounds of 8, set it explicitly to the smallest number your task actually needs, and run a small batch first to see the real average round count before scaling up.

EndpointsCopy Anchor Link

Input formatCopy Anchor Link

Each line of the input file is one request object, matching the OpenAI batch input shape:

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gemma-4-31b", "messages": [{"role": "user", "content": "Name three primary colors."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gemma-4-31b", "messages": [{"role": "user", "content": "Name three prime numbers."}]}}

Tool calling in a batch (openrelay.tool_config)Copy Anchor Link

A /v1/chat/completions record may carry standard OpenAI tools / tool_choice fields plus one OpenRelay extension, tool_config, that tells the batch worker how to execute the model's tool calls between turns. It lives under the reserved openrelay object on the body, the same namespace the response side uses for tool_rounds:

"openrelay": {
  "tool_config": {
    "url": "https://your-app.example.com/api/openrelay-tools",
    "authorization": "Bearer <token>",
    "context": { "contractId": "..." },
    "max_rounds": 8,
    "timeout_ms": 30000
  }
}

// before (no longer accepted)
"body": { "model": "...", "messages": [], "tool_config": { "url": "..." } }
// after
"body": { "model": "...", "messages": [], "openrelay": { "tool_config": { "url": "..." } } }

Omitted or 0 values for max_rounds and timeout_ms take the defaults, and values above the caps are clamped down to them; a negative value rejects the record when you create the batch.

When a turn finishes with finish_reason: "tool_calls", the worker POSTs { "custom_id", "context", "tool_calls" } to url and expects { "results": [{ "tool_call_id", "content" }] } back. It appends the assistant message and one role: "tool" message per result, then re-invokes the model, up to max_rounds times.

• openrelay.tool_config is stripped before every model invocation and never appears in output or error files. If openrelay carries nothing else, the whole object is removed too, so the model sees exactly the body you would have written by hand.
• The final output line's usage sums prompt and completion tokens across all turns of the record (that is also what you are billed on), and the body carries "openrelay": { "tool_rounds": N } when any round ran.
• An executor failure (non-2xx, timeout, or a malformed response) fails that record with code tool_execution_failed; a malformed tool_config rejects it at create with invalid_tool_config, and the retired bare spelling rejects it with tool_config_moved.
• If max_rounds is exhausted, the last model response is stored as a normal output line with finish_reason: "tool_calls", matching OpenAI.
• Records without openrelay.tool_config behave exactly as before, even when they carry tools.
• Make your executor idempotent. Batch processing retries and resumes internally, so a record's tool calls can be executed more than once (the same tool_call_id may hit your endpoint again). Executors must be idempotent or tolerate duplicate calls, and a retried multi-round record's stored usage reflects the run that produced its final output.
• A record that fails mid-loop (for example an executor error on round 3) produces an error line, and the tokens its earlier turns consumed are not billed.

MCP executors ("type": "mcp")Copy Anchor Link

Set type to "mcp" to back a record's tools with any MCP server speaking Streamable HTTP (JSON-RPC 2.0 over a single POST endpoint, protocol revision 2025-03-26 or later), instead of implementing the plain POST protocol above:

"openrelay": {
  "tool_config": {
    "type": "mcp",
    "url": "https://your-mcp-server.example.com/mcp",
    "authorization": "Bearer <token>",
    "context": { "contractId": "..." }
  }
}

• Once per record loop (cached across rounds) the worker sends initialize (with clientInfo.name openrelay-batch-worker) followed by notifications/initialized, honoring any Mcp-Session-Id the server issues. Stateless servers that answer initialize with an HTTP 404 or 405 (no handshake endpoint) are handled: the worker falls back to sending tools/call directly. Any other failing status, including 401 and 403, fails the record rather than silently proceeding unauthenticated.
• Each model tool call becomes one tools/call request whose name is the function name and whose arguments is the parsed JSON object from the model's arguments string. Responses may be application/json or text/event-stream (the final JSON-RPC message of the stream is used).
• The tool result delivered back to the model is the concatenation of the result's content items of type "text". A result with isError: true is not a record failure: its text is delivered to the model as the tool content, matching MCP semantics, so the model can react to it.
• MCP tools/call has no side channel for context, so when openrelay.tool_config.context is present it is merged into every call's arguments under the reserved _context key (omitted entirely when context is absent). Don't define a tool parameter named _context.
• Transport errors, non-2xx responses, and JSON-RPC error responses fail the record with tool_execution_failed, exactly like the http type.
• Everything else is unchanged from the http type: authorization is sent on every request, timeout_ms bounds each request, max_rounds bounds the loop, and the same idempotency advice applies.

Dependent requests in a batch (openrelay.workflow)Copy Anchor Link

When later requests depend on earlier answers, a record can carry openrelay.workflow instead of running once: your callback returns the next requests, OpenRelay runs them inside the record and posts the results back, and the record ends when the callback answers completed or failed. Requests returned in one step run concurrently, the record is bounded by steps, requests, and tokens rather than a clock, and every request that succeeds is billed. Enabled per organization. See Workflows.

PricingCopy Anchor Link

Batch requests are metered per token exactly like Sync (real-time) requests. A standard batch, the default, bills 50% of the Sync rate on input, cached input, and output. An expedited batch bills the Sync rate; see Priority. A batch's rolled-up token counts and cost appear in the usage object on the batch as it runs. See Pricing & billing.

GuidesCopy Anchor Link