Overview
Submit inference requests as a job, delivered within 1 hour or 24 hours at a discount from Sync. Read results as they finish or get them by webhook.
Async is in beta
The API, its job and result shapes and the pages here can still change before general availability. Pin your code to the fields documented here.
The Async API runs inference requests that do not need an answer on an open connection. You submit a job of one to many requests, pick how long we have to deliver it, and read the results as they finish or get them pushed to a webhook when the job ends. In return the requests cost less than Sync.
| Window | Delivered | Price |
|---|---|---|
1h | Within 1 hour of the create | 20% off Sync |
24h | Within 24 hours of the create | 50% off Sync |
A request that has not started when its window ends expires and costs nothing. A request that started finishes and is charged. See Pricing.
What it takes
- A job is the unit. A single request is a job of one. Create, list, get, cancel and results all work on the job.
- The request lines are the Batch API's. Each line is
{custom_id, method, url, body}, withurlrequired:/v1/chat/completions,/v1/messagesor/v1/embeddings. A job can mix models and endpoints. A line that asks to stream is rejected. - Models. A line must name a model that is offered for Async in the job's
window. A line that does not is rejected with
model_not_offered, and its message lists the models offered for that window. - Results while the job runs. Results arrive a group of requests at a time. You can page through them before the job ends.
Base URL
Every Async route is on inference.openrelay.inc, under /v1, with the
same or_ API key and Authorization: Bearer header as the rest of
inference. api.openrelay.inc does not
serve them.
| Route | Purpose |
|---|---|
POST /v1/async/jobs | Create a job |
GET /v1/async/jobs | List your organization's jobs, newest first |
GET /v1/async/jobs/{id} | One job, its counts and its cost so far |
POST /v1/async/jobs/{id}/cancel | Stop starting the job's requests |
GET /v1/async/jobs/{id}/results | Finished results, a page at a time |
The API reference has every field. The orl CLI has a command
for each route under orl async.
Workflows
Run dependent inference inside one batch record. Your callback returns the next requests and OpenRelay runs them, bounded by steps, requests, and tokens.
Quickstart
Create an async job with curl, the orl CLI or the OpenAI Python SDK, watch it finish, and read the results of the requests it ran.