Skip to content
OpenRelay is in early access, and the /v1 API is stable. New capabilities ship in the changelog.
Skip to Content
GuidesInference APIsAsync API

Agents

What an async request can do for an agent today. Tool calls come back for your own loop; OpenRelay-run tool loops and workflows are not available in Async.

Async is in beta

Shapes and limits can still change before general availability.

Each line of an async job is one model call, answered the way the Sync endpoint would answer it.

What works

  • tools and tool_choice in a line's body are passed to the model. If it decides to call a tool, the result's response.body carries the tool call, as it would on a Sync response.
  • Your own loop. Run the tool yourself, then add the next turn, with the tool result in its messages, to a new job. For many agent runs in parallel, put one line per run in a job and make the next job from the results of the last.

What does not

openrelay.tool_config (OpenRelay runs the tool loop) and openrelay.workflow (your callback decides each step) work in the Batch API but are not supported in an async job. A line that carries either is rejected when the job is created with invalid_body, is never run and is never billed. Use the Batch API for those today.

Streaming is not supported either: a line with "stream": true is rejected with streaming_unsupported.

On this page