Workflows

A workflow record is a batch record whose requests are decided by your own endpoint as the record runs. OpenRelay posts an event to your callback, runs the inference requests the callback returns, posts their results back, and repeats until the callback answers completed or failed. Use it when later requests depend on earlier answers (extract, then reconcile, then summarize) and you want all of that work to stay inside one batch, at the batch rate.

Every dependent request runs inside the original record, on the batch's model, and never creates another batch. The record produces one result line: your callback's final output, or an error.

The recordCopy Anchor Link

A workflow record is a /v1/chat/completions record whose body carries openrelay.workflow, the same reserved openrelay object that holds tool_config:

{"custom_id": "contract-42", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openrelay/gemma-4-31b", "messages": [{"role": "user", "content": "contract-42"}], "openrelay": {"workflow": {"url": "https://your-app.example.com/api/openrelay-workflow", "authorization": "Bearer <token>", "context": {"contractId": "contract-42"}, "max_steps": 16, "max_requests": 64, "max_tokens": 2000000, "callback_timeout_ms": 60000}}}}

The record's own messages are not sent to the model. The first requests come from your callback's answer to the start event, so put whatever the callback needs to begin in context.

Omitted fields take their defaults. A value outside its range, a non-HTTPS url, or a missing authorization fails that record at validation with invalid_workflow, before any capacity is provisioned.

The callbackCopy Anchor Link

OpenRelay sends POST <url> with your authorization value and a JSON body:

Each entry of results:

Answer 200 with one of four states:

Each entry of requests is one inference:

A request body may name only the batch's model (or omit model), cannot stream, and cannot carry openrelay.workflow. It may carry openrelay.tool_config, and its tool rounds count as that one request. A request that breaks these rules does not run: it comes back in results with status 400.

Your answer can be up to 4 MiB. A 429, a 5xx, a timeout, or a connection error is retried, up to four attempts per event with callback_timeout_ms on each. Any other non-2xx answer fails the record with workflow_callback_rejected, and an answer that breaks the rules above fails it with workflow_invalid_response.

BudgetsCopy Anchor Link

A record is bounded by the work it does, never by a per-record clock (each request it runs has its own time limit, see Time limits):

• Steps. Every continue or wait answer is one step. The limit is checked after the answer, so your callback can still answer completed or failed on its last step. A wait pauses at most 60 seconds, so a record that only waits stops after about max_steps minutes.
• Requests. Before a step runs, OpenRelay adds its requests to the ones the record has already run. If that passes max_requests, none of the step's requests run and the record fails. Every request counts, including one that comes back 400 or 500.
• Tokens. Prompt plus output tokens, summed over the record's requests. This is a record budget, separate from the max_tokens a request body sets for one completion. When a finished request takes the total past the limit, the step's requests still running are cancelled and not billed, and the record fails. The total can end slightly above the limit, by the requests that finished in that step.

ConcurrencyCopy Anchor Link

• Requests in one step run concurrently. Return independent requests in the same step to run them side by side.
• Steps run in order. The next event is sent only when every request of the step has finished, and its results are in the order you returned the requests.
• Capacity is per model, not per record. OpenRelay sets how many requests run at once for each model, and every record in the batch shares that capacity. Returning 32 requests in a step does not reserve 32 slots; they run as slots free up, alongside other records. A record that is waiting on your callback or on a wait holds no model capacity.

BillingCopy Anchor Link

Every request that returns a successful response is billed at the batch rate, on the tokens the model reported for it, when it finishes. That holds even if the record later fails, hits a budget, or expires: you pay for the requests that ran, and nothing for the ones that did not. Requests that come back with an error, and requests cancelled by max_tokens or the end of the batch window, are not billed.

The record's result line carries a usage object summed over its requests, and the batch's usage includes the tokens and cost of every workflow record, failed and expired ones included. request_counts still counts records.

Time limitsCopy Anchor Link

A workflow record has no deadline of its own: it can run for as long as it keeps doing work, up to its budgets, inside the batch's 24-hour window. When the window ends, a record that has not finished gets an error line with batch_expired. Requests it completed before then stay billed; requests still running are cancelled.

Each request the record runs has the batch's 30-minute limit per model call. A request that has not returned its response within 30 minutes is stopped, is not retried, and is not billed. It comes back to your callback with status_code 500 and an error that starts with request_time_limit, and the record keeps running, so your callback decides what to do next. A request whose model server failed under it is the exception: it runs again on new capacity.

callback_timeout_ms bounds one HTTP call to your callback and nothing else. If all four attempts to deliver one event fail, the record fails with workflow_callback_unreachable, keeps the bill for the requests that ran, and the rest of the batch keeps running.

Result and failure codesCopy Anchor Link

A completed record's output line has status_code 200 and your output as its body, with a usage object added:

{
  "id": "batch_req_contract-42",
  "custom_id": "contract-42",
  "response": {
    "status_code": 200,
    "body": {
      "summary": "Net 30, auto-renews yearly, 90-day termination notice.",
      "usage": { "prompt_tokens": 35595, "completion_tokens": 416, "total_tokens": 36011 }
    }
  },
  "error": null
}

A record that fails gets an error line with one of these codes:

Worked exampleCopy Anchor Link

A contract record extracts three clauses in parallel, then asks for a summary built from them.

1. Start. OpenRelay sends the first event:

{
  "event_id": "wf_evt_3c1d9a0e5b7f42c8a6e1d0b9f4c2a7e3",
  "event": "start",
  "workflow_id": "batch_9f8e7d6c5b4a:contract-42",
  "record_id": "contract-42",
  "step": 0,
  "attempt": 1,
  "context": { "contractId": "contract-42" }
}

The callback returns three independent requests, which run concurrently:

{
  "state": "continue",
  "requests": [
    { "id": "payment", "custom_id": "payment", "body": { "model": "openrelay/gemma-4-31b", "messages": [{ "role": "user", "content": "Extract the payment terms: ..." }] } },
    { "id": "renewal", "custom_id": "renewal", "body": { "model": "openrelay/gemma-4-31b", "messages": [{ "role": "user", "content": "Extract the renewal terms: ..." }] } },
    { "id": "termination", "custom_id": "termination", "body": { "model": "openrelay/gemma-4-31b", "messages": [{ "role": "user", "content": "Extract the termination terms: ..." }] } }
  ]
}

2. Results. When all three finish, the callback gets them in the same order (bodies shortened):

{
  "event_id": "wf_evt_8b2e4f6a1c3d5e7f9a0b2c4d6e8f0a1b",
  "event": "results",
  "workflow_id": "batch_9f8e7d6c5b4a:contract-42",
  "record_id": "contract-42",
  "step": 1,
  "attempt": 2,
  "context": { "contractId": "contract-42" },
  "results": [
    { "id": "payment", "stage_id": "payment", "attempt": 1, "custom_id": "payment", "status_code": 200, "body": { "choices": [{ "message": { "content": "Net 30." } }] }, "usage": { "input_tokens": 11820, "output_tokens": 140 } },
    { "id": "renewal", "stage_id": "renewal", "attempt": 1, "custom_id": "renewal", "status_code": 200, "body": { "choices": [{ "message": { "content": "Auto-renews yearly." } }] }, "usage": { "input_tokens": 11790, "output_tokens": 96 } },
    { "id": "termination", "stage_id": "termination", "attempt": 1, "custom_id": "termination", "status_code": 200, "body": { "choices": [{ "message": { "content": "90 days written notice." } }] }, "usage": { "input_tokens": 11805, "output_tokens": 118 } }
  ]
}

The callback returns one dependent request:

{
  "state": "continue",
  "requests": [
    { "id": "summary", "custom_id": "summary", "body": { "model": "openrelay/gemma-4-31b", "messages": [{ "role": "user", "content": "Summarize: Net 30. Auto-renews yearly. 90 days written notice." }] } }
  ]
}

3. Done. On the next results event (step 2), the callback answers:

{ "state": "completed", "output": { "summary": "Net 30, auto-renews yearly, 90-day termination notice." } }

The record used 2 steps, 4 requests, and 36,011 tokens, well inside the budgets in its config, and produced the output line shown above. All four requests are billed. Had the callback answered failed at step 2 instead, the record would get a workflow_failed error line and all four requests (the three extractions and the summary, which had already run) would still be billed.