Errors
How the OpenRelay API reports errors: the JSON error body, status codes, when to retry, idempotent creates, and capacity and balance refusals.
The API uses conventional HTTP status codes. 2xx means success. 4xx means
the request was rejected (and usually shouldn't be retried unchanged). 5xx
means something went wrong on our side.
Error body
Every error response has a JSON body with a human-readable error message and
an optional machine-readable code:
{
"error": "vm not found",
"code": "not_found"
}Always branch on the HTTP status code first; treat code as a stable hint
and error as a message for logs and humans. The one exception is 503, where
the code tells a capacity refusal from a fault (see
Capacity refusals).
Status codes
| Status | Meaning | What to do |
|---|---|---|
200 | OK | Success. |
400 | Bad Request | Malformed JSON or invalid parameters. Fix the request. |
401 | Unauthorized | Missing, invalid, or expired API key. Check the Authorization header. |
403 | Forbidden | The key or member lacks the permission or targets another org (see Permissions), or the account's balance or a review refuses the action (see Account and balance refusals). |
404 | Not Found | The resource doesn't exist (or isn't visible to your org). |
409 | Conflict | The request conflicts with durable current state (e.g. a name clash, an action invalid for the resource's status, a feature that is switched off, FEATURE_DISABLED, or a VM stop while stopping is switched off, STOP_DISABLED). Fix the request; retrying it unchanged will keep failing. The one exception is IDEMPOTENCY_KEY_IN_USE: a create with the same Idempotency-Key is still running, so repeat yours a second later (see Idempotent creates). |
422 | Unprocessable | The request is well-formed but semantically invalid. IDEMPOTENCY_KEY_REUSED: the Idempotency-Key was used for a create with a different body (see Idempotent creates). |
429 | Too Many Requests | You're being rate limited. Back off and retry; honor Retry-After. |
503 | Service Unavailable | With a code from the Capacity refusals table, there is no capacity for the request right now: pick another shape or try later, and don't repeat the same request in a loop. Any other 503 is a transient problem on our end: retry with backoff. |
5xx | Server Error | A transient problem on our end. Retry with backoff. |
Retrying
Retry safely
Retry only idempotent requests (GET, and a create you send with an
Idempotency-Key: see Idempotent creates) and only on
429 and 5xx. Use exponential backoff with jitter, and
respect the Retry-After header on 429 responses. A 503 carrying a
capacity code is the exception: see Capacity refusals.
A 409 is a durable conflict (a name clash, or a node you targeted being
offline or on hold); a request that is structurally impossible (a GPU count
your fleet cannot allocate) is a 400. Retrying a 409 or 400 unchanged
will not help. Branch on the status code, not the error code, except
to tell a capacity 503 from any other, and an IDEMPOTENCY_KEY_IN_USE
409, which a repeat after Retry-After clears.
# Example: GET a VM, retrying on 429, a 5xx or a dropped connection, up to
# 5 attempts with exponential backoff and jitter. Returns non-zero on failure.
get_vm() {
local attempt delay status
for attempt in 1 2 3 4 5; do
status=$(curl -sS -o vm.json -w '%{http_code}' \
"https://api.openrelay.inc/v1/vms/$VM_ID" \
-H "Authorization: Bearer $OPENRELAY_API_KEY")
case "$status" in
2??) return 0 ;;
429|5??|000) ;; # retryable
*) echo "GET failed with HTTP $status" >&2; return 1 ;;
esac
if [ "$attempt" -lt 5 ]; then
delay=$((1 << attempt)) # 2, 4, 8, 16 seconds
sleep "$((delay + RANDOM % delay))" # plus up to as much again in jitter
fi
done
echo "GET still failing after 5 attempts (HTTP $status)" >&2
return 1
}
get_vm || exit 1Idempotent creates
A create that timed out or lost its connection may still have gone through, so
repeating it can make a second VM. Send an Idempotency-Key header, a value
you choose once per create (1 to 512 bytes, such as a UUID), and send the same
key and the same body on every retry: the create runs once.
curl -X POST "https://api.openrelay.inc/v1/orgs/$ORG_ID/vms/create" \
-H "Authorization: Bearer $OPENRELAY_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: 3f6c1d2e-8a4b-4c3d-9e1f-5a6b7c8d9e0f" \
-d '{"name":"cuda-devbox","gpuModelId":"rtx-4090","gpuCount":1,"sshKeyIds":["key_2f9c1a"]}'| A request with the key | The response |
|---|---|
| The first one | Runs. A 2xx answer is kept for 24 hours. |
A repeat with the same body, after the first answered 2xx | That answer again, with the same status and the header Idempotent-Replayed: true. Nothing new is created or billed. |
| A repeat with the same body, while the first is still running | 409 IDEMPOTENCY_KEY_IN_USE, with Retry-After: 1. Repeat it a second later. |
| A repeat with the same body, after the first succeeded but its answer could not be kept (one over 400 KB) | 409 IDEMPOTENCY_KEY_USED. Nothing runs. Find what the first request created (list your VMs or pods) instead of repeating it. |
| The same key with a different body | 422 IDEMPOTENCY_KEY_REUSED. Nothing is created. Use a new key for a new create. |
A repeat after the first answered anything but 2xx | Runs again: only a successful answer is kept, so a retry after a 400, a 429 or a 5xx is a new attempt. One exception: a batch create that answered 503 orchestrator_unavailable because we could not tell whether its batch started holds the key for up to two minutes first (409 IDEMPOTENCY_KEY_IN_USE, lower case on the inference host). That batch does not run, unless we also failed to record that it failed; list your batches before you create again if you need to be sure. |
| A key that is empty or longer than 512 bytes | 400 INVALID_IDEMPOTENCY_KEY. The header sent twice is a 400 BAD_REQUEST. |
It applies to these creates:
| Operation | Codes |
|---|---|
POST /v1/orgs/{orgId}/vms/create | IDEMPOTENCY_KEY_IN_USE, IDEMPOTENCY_KEY_USED, IDEMPOTENCY_KEY_REUSED, INVALID_IDEMPOTENCY_KEY |
POST /v1/pods | the same |
POST /v1/batches (on inference.openrelay.inc) | idempotency_key_in_use, idempotency_key_used, idempotency_key_reused, invalid_idempotency_key (also for the header sent twice), in the inference error body |
- Keys are scoped to your organization and the operation. Two API keys of one organization share them, and the same key sent to a VM create and to a pod create makes one of each.
- What counts as the same body. A VM or pod create compares the fields it reads: the same fields and values in another order or with other whitespace are the same request, and a field the API does not know is ignored. A batch create compares the bytes you send, so keep the bytes of the first attempt.
- A repeat is still authorized. It answers only to a caller that may make
the create (a read-only key gets its
403, not the stored answer); a batch create also passes the balance check first. - A request still running after its process stopped holds the key for at most two minutes; a repeat after that runs the create again.
- The key is counted in bytes, not characters: 512 ASCII characters fit, fewer characters of a multibyte script do.
- File uploads (
POST /v1/files) take no key: a repeated upload stores a second file, which costs nothing until a batch runs it.
Capacity refusals
A 503 with one of the codes below means the request is valid, but there is
no capacity for it right now. Sending the same request in a loop does not make
room. These codes read live fleet availability, so they can clear as nodes and
VMs turn over. The Retry-After header on these responses is deprecated: don't
depend on it.
| Code | Returned by | What to do |
|---|---|---|
NODE_AT_CAPACITY | VM create or fork with nodeId | The node you named has no free GPUs of that count. Drop nodeId, pick another node or count, or try later. |
INSUFFICIENT_GPU_CAPACITY | VM create or fork | No node has that many GPUs of the model free. Pick another GPU model or count, or try later. |
INSUFFICIENT_GPU_CAPACITY | pod create | No node has gpuCount GPUs of the model free for you: a pod's GPUs all come from one node. Pick another GPU model or a smaller gpuCount, or try later. |
INSUFFICIENT_CPU_CAPACITY | VM create or fork | Not enough free CPU cores for the size. Pick a smaller size, or try later. |
INSUFFICIENT_HOST_CAPACITY | GPU VM create or fork | A node's share covers the shape, but no node has it free. Pick a smaller shape, or try later. |
INSUFFICIENT_HOST_CAPACITY | pod create | The nodes that fit cannot size a pod's RAM and CPU yet, usually because they are being brought up and have not reported their host inventory. Try again in a few minutes. |
INSUFFICIENT_IP_CAPACITY | VM create or fork, pod create or start | No free guest network address, or for an InfiniBand pod no free InfiniBand address on the pod's fabric. Terminate unused workloads, or try later. |
VM_NODE_BUSY | restart of a VM | For a stopped or failed VM: the node holding it has too few free GPUs of its model, a GPU the VM needs has failed, or the node has too little disk, RAM or CPU, to start it. For a running VM: a GPU it runs on has failed. The message names which, including the failed GPU's address, and says when stopping the VM and starting it again would place it on healthy GPUs of its node. Try later, or create a new VM to place elsewhere. |
POD_NODE_BUSY | restart of a pod | For a stopped or failed pod: the node holding the pod's volume has no free GPUs of its model. For a running pod: a GPU it runs on has failed. The message names it, and says when stopping the pod and starting it again would place it on healthy GPUs of its node. Try later, or delete the pod and create a new one. |
The restart codes (VM_NODE_BUSY, POD_NODE_BUSY, and INSUFFICIENT_IP_CAPACITY
on a pod start) behave differently from the create codes. A stopped VM or pod is
pinned to the node that holds its disk or volume, so a different shape does not
help: the refusal clears when that node frees capacity.
Account and balance refusals
The codes below refuse a request because of the account, not the request or
the key: the balance is too low, the account is under review, or it is being
verified. All of them are 403s except TOO_MANY_PAYMENT_ATTEMPTS, which is a
400. A new API key or a wider scope clears none of them. In the table, a
new workload is a VM or pod create, or a VM fork.
| Code | Returned by | What to do |
|---|---|---|
INSUFFICIENT_BALANCE | new workloads, and a restart or reboot of one you have | The available balance is under $0.50, or it dropped below $0.50 and the account was paused, which stops running workloads. Add funds or turn on auto-recharge (VM billing). Stopped workloads stay stopped: start them again once the balance is back. |
ACCOUNT_RESTRICTED | new workloads, API key create | The account is under review. Running workloads, restarts, deposits and existing keys keep working. Contact support. |
ACCOUNT_PENDING_VERIFICATION | new workloads, API key create, deposit, card save, runner pool create | The account is being verified, which usually takes under one business day. Running workloads, restarts and existing keys keep working; deposits and auto-recharge are paused until it is done. You get an email when it is, and you can tell us about your use case from the banner in the dashboard. |
VERIFICATION_UNAVAILABLE | deposit or card save that carries device identification (the dashboard's do) | The payment could not be verified, and the message says which case applies. If it asks you to try again, retry in a moment and contact support if it keeps failing. If it asks you to contact support, the account needs a review, and retrying will not change the answer. |
FUNDING_UNDER_REVIEW | deposit, card save | The account is under review and cannot add funds. Contact support. |
TOO_MANY_PAYMENT_ATTEMPTS | deposit | Too many payments failed in the last hour or two. The refusal clears on its own within two hours of the last failure. The cooldown does not refuse a card save, so you can add a card that works. |
ORG_SUSPENDED | any request for the organization, and organization create | The organization is suspended pending review, and its running workloads are stopped. On organization create, it means an organization you own or administer is suspended. Contact support. |
ORG_DISABLED | any request for the organization, and organization create | The organization has been deleted. On organization create, it means an organization you own or administer has been deleted. Contact support. |
The account is checked before the balance: a new workload on an account that
is being verified and whose available balance is under $0.50 is refused with
ACCOUNT_PENDING_VERIFICATION, because adding funds would not lift the hold.
ORG_SUSPENDED and ORG_DISABLED are returned when the request is
authenticated, before it reaches the action, so any authenticated endpoint can
return them.
Two more 403s are about the signed-in user rather than the organization.
EMAIL_UNVERIFIED refuses every request made with a dashboard session whose
email address is not confirmed: confirm it from the confirmation email, then
sign in again. API keys are not affected. ORG_LIMIT refuses an organization
create when you already own five organizations; contact support if you need
another. An account creates at most one organization every ten seconds, and a
create inside that window answers 429 ORG_CREATE_IN_PROGRESS: retry after
Retry-After.
A card save can also end without the card. Your bank may be asked to confirm a
new card, and if it does not, the card is removed and the save answers
CARD_NOT_VERIFIED: add a different card. A card form started while a newer
one was opened for the same account answers 503 SETUP_INTENT_SUPERSEDED:
start the card form again. A second card the bank does not
confirm puts the account on verification (ACCOUNT_PENDING_VERIFICATION).
The inference API refuses billable requests from a restricted account, or one being verified, with its own codes: see Inference errors.
Permissions
In an organization with permissions enabled, a 403
that a role, key restriction or policy decided names the action it needed,
the layer that refused it, and the deciding policies, and the
OpenRelay-Required-Permission header carries the action:
{
"error": "This API key is limited and does not include compute.create (Compute: Manage).",
"code": "FORBIDDEN",
"permission": "compute.create",
"deniedBy": "key",
"policies": ["key:k_9f2:permissions"]
}deniedBy is role (the role does not grant it), key (the key's
restrictions), org_policy (a guardrail or Cedar policy of your
organization) or platform (a platform rule, such as another organization's
resource or an action only a signed-in session may take). Ask an admin for a
role or key that grants the permission, or explain the decision with the
access checker. Without permissions
enabled, a 403 FORBIDDEN names no permission: see
Legacy scopes.
| Code | Status | Returned by | What to do |
|---|---|---|---|
ESCALATION_REFUSED | 403 | key and role writes, member role changes | You granted a permission or model you do not hold yourself; permission names the first one. Grant less, or ask someone who holds it. |
JWT_REQUIRED | 401 | creating an organization, accepting or declining an invitation, profile changes or device signals, called with an API key | The operation takes a signed-in session. Key management, members, roles and policies refuse an API key with 403 and deniedBy: platform instead. |
KEY_LIFETIME | 400 | key create | Your organization requires keys to expire: set expiresInSeconds within its maximum lifetime. |
ROLE_IN_USE | 409 | role delete | The role is assigned. The body lists the members that hold it; change their roles first. |
WOULD_LOCK_OUT | 409 | policy create or update | Saving would take policy or member management away from you. Add your network or role to the rule first. |
POLICY_INVALID | 400 | policy create or update | The policy did not parse, validate or stay inside your organization. diagnostics lists each problem with its line and column. |
The inference API reports a permissions denial in its own error shape, 403
with type: permission_error: code permission_denied (with permission and
denied_by), or model_not_allowed (with model and denied_by) when the
model is outside the key's, role's or organization's model list.
Sizing a GPU VM
A GPU VM is sized from the host capacity its GPUs come with, not from a machine tier. Three codes are specific to that:
| Code | Status | Meaning |
|---|---|---|
RESOURCE_SIZE_NOT_APPLICABLE | 400 | You sent resourceSize on a create with gpuCount above zero. That field is the CPU-only VM tier picker. Drop it, and size the VM with diskSizeGb (and, where available, the guest RAM field) or omit both for auto. |
VM_SHAPE_NOT_PLACEABLE | 400 | No node's per-GPU share, times this GPU count, covers the guest RAM or disk asked for. The message carries the largest share on offer. Lower the request, choose auto, or take more GPUs; retrying unchanged will not help. |
INSUFFICIENT_HOST_CAPACITY | 503 | Some node's share covers the shape, but no node has it free right now. Pick a smaller shape, or try later (Capacity refusals). |
An explicit request is capped at one node's per-GPU share times the GPU count, so a one-GPU VM cannot take a multi-GPU node's whole host. Auto sizing takes the tightest node that can still give a full share, which keeps RAM-rich nodes available to larger counts. The floor is 65% of the VRAM the VM is placed against for guest RAM. The boot disk defaults to 100 GB per GPU and its only floor is the flat platform minimum, so a smaller disk is a valid request.
Diagnosing failures
For resources that can land in a failed or stuck state (VMs and pods), the
resource body carries a statusReason field with a human-readable cause: read
it before retrying a create.