Quickstart
Run a batch end to end. Build a JSONL file, upload it, create a batch, poll to completion, and download the results, with curl, the orl CLI, or the OpenAI SDK.
This walks through a full batch three ways: raw curl, the
orl CLI, and the OpenAI Python SDK (the API is OpenAI-compatible,
so the SDK works unchanged with a different base URL). Every request here goes
to inference.openrelay.inc, the inference host; api.openrelay.inc is
the control plane and does not serve batch routes. Set your API key first:
export OPENRELAY_API_KEY="or_your_api_key"CLI users can run orl auth login once instead; it stores the key for every
later command.
Build a JSONL input file
One request per line. Each line needs a custom_id you choose (echoed back on
the result), and a body that is exactly what you would POST to
/v1/chat/completions online.
cat > requests.jsonl <<'EOF'
{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gpt-oss-120b", "messages": [{"role": "user", "content": "Name three primary colors."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gpt-oss-120b", "messages": [{"role": "user", "content": "Name three prime numbers."}]}}
EOFUpload the file and create the batch
One command uploads the file and submits the batch
(completion_window defaults to 24h):
orl batches create --input-file requests.jsonl --endpoint /v1/chat/completionsuploaded requests.jsonl as file-a1b2c3d4e5f6a7b8c9d0e1f2
{
"id": "batch_9f8e7d6c5b4a",
"status": "validating",
...
}To upload first and submit separately, use orl files upload requests.jsonl
and pass the returned id with --input-file-id.
For small jobs you can skip the upload and pass the requests inline with a
requests array instead of input_file_id. See
Batches.
Poll until it finishes
Fetch the batch until status is terminal (completed, failed, expired,
or cancelled). request_counts fills in as work progresses.
orl batches get batch_9f8e7d6c5b4aorl batches list shows your recent batches at a glance, and
orl batches cancel <id> stops one you no longer need.
{
"id": "batch_9f8e7d6c5b4a",
"status": "completed",
"output_file_id": "file-batch_9f8e7d6c5b4a-output",
"request_counts": { "total": 2, "completed": 2, "failed": 0 },
"usage": { "input_tokens": 34, "output_tokens": 58, "cost_nano_usd": 41000 }
}Download the results
The output_file_id holds the successful responses; if any requests failed, an
error_file_id holds those. Both are JSONL, one line per request, joined to
your inputs by custom_id.
orl files content get file-batch_9f8e7d6c5b4a-output -o results.jsonlEach line pairs your custom_id with the response. See
Results for the exact shape.
Next steps
- Files: presigned uploads for large files, and reading file metadata and content.
- Batches: the full batch object, inline requests, listing, and cancellation.
- Results: the output and error file formats.
- CLI reference: install
orland authenticate once withorl auth login.
Overview
The OpenRelay Batch API. Submit large jobs of chat or text completions as a JSONL file, run them asynchronously within 24 hours, and download the results at a reduced per-token rate.
Files
Upload batch input files as JSONL (a direct multipart upload for smaller files, a presigned URL for large ones), then read file metadata and content.