Skip to content
OpenRelay is in early access, and the /v1 API is stable. New capabilities ship in the changelog.
Skip to Content
GuidesInference APIsAsync API

Overview

Submit inference requests as a job, delivered within 1 hour or 24 hours at a discount from Sync. Read results as they finish or get them by webhook.

Async is in beta

The API, its job and result shapes and the pages here can still change before general availability. Pin your code to the fields documented here.

The Async API runs inference requests that do not need an answer on an open connection. You submit a job of one to many requests, pick how long we have to deliver it, and read the results as they finish or get them pushed to a webhook when the job ends. In return the requests cost less than Sync.

WindowDeliveredPrice
1hWithin 1 hour of the create20% off Sync
24hWithin 24 hours of the create50% off Sync

A request that has not started when its window ends expires and costs nothing. A request that started finishes and is charged. See Pricing.

What it takes

  • A job is the unit. A single request is a job of one. Create, list, get, cancel and results all work on the job.
  • The request lines are the Batch API's. Each line is {custom_id, method, url, body}, with url required: /v1/chat/completions, /v1/messages or /v1/embeddings. A job can mix models and endpoints. A line that asks to stream is rejected.
  • Models. A line must name a model that is offered for Async in the job's window. A line that does not is rejected with model_not_offered, and its message lists the models offered for that window.
  • Results while the job runs. Results arrive a group of requests at a time. You can page through them before the job ends.

Base URL

Every Async route is on inference.openrelay.inc, under /v1, with the same or_ API key and Authorization: Bearer header as the rest of inference. api.openrelay.inc does not serve them.

RoutePurpose
POST /v1/async/jobsCreate a job
GET /v1/async/jobsList your organization's jobs, newest first
GET /v1/async/jobs/{id}One job, its counts and its cost so far
POST /v1/async/jobs/{id}/cancelStop starting the job's requests
GET /v1/async/jobs/{id}/resultsFinished results, a page at a time

The API reference has every field. The orl CLI has a command for each route under orl async.

On this page