Decisions

Answer typed questions about a piece of input with a classifier instead of a generator. You send a state (the text or JSON to decide about) and a set of questions; each answer comes back as a label or a score with its probabilities and a confidence. Nothing is generated, so only input tokens are billed.

POST https://inference.openrelay.inc/v1/decisions

The model that serves this family is openrelay/laya, built on Laya by Convai Innovations (Apache 2.0): an encoder-only decision model with an English checkpoint and a multilingual one. Check a model's supported_apis on GET /v1/models for decisions; a model that does not list it answers 400 unsupported_api. Decisions are online only: the Batch API does not accept /v1/decisions.

Question types

Every question has one of three types:

Request

Each question:

Any other field is rejected with 400 unknown_field, at the top level, inside a question, or inside noul criteria. That includes checkpoint, task, and lang: there is no way to pick a checkpoint in the request. stream is dropped: a decision is always a single JSON response.

Use question ids made of ASCII letters, digits, _, and -. They come back as the keys of answers, and an error names them in its param field (see Errors).

Checkpoint routing

Each request is routed by the script and language of its state. Text in a non-Latin script goes to the multilingual checkpoint, and so does Latin-script text detected as a language other than English. Everything else goes to english: English text, a state with no letters at all, and Latin-script text whose language is not identified and which has too few non-English letters to tell. The response reports which checkpoint answered in routing.checkpoint, and why in routing.reason. The reason is text for your logs; do not parse it. Because the option limit depends on the checkpoint, read routing.checkpoint rather than assuming it from the input language.

Response

{
  "model": "openrelay/laya",
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": { "billing": 0.9731, "technical": 0.0269 },
      "confidence": 0.8217,
      "action": { "act_probability": 1.0 }
    },
    "urgency": {
      "type": "score",
      "score": 1.6993,
      "legend": { "0": "not urgent", "1": "soon", "2": "blocking" },
      "probabilities": { "0": 0.0499, "1": 0.2009, "2": 0.7492 },
      "confidence": 0.3735,
      "action": { "act_probability": 1.0 }
    },
    "churn_risk": {
      "type": "noul",
      "noul": 0.8244,
      "confidence": 0.8244,
      "action": { "act_probability": 1.0 }
    }
  },
  "routing": { "checkpoint": "english", "reason": "English Latin text" },
  "usage": { "prompt_tokens": 173, "total_tokens": 173, "truncated_tokens": 0 }
}

Limits

The option limit is checked after routing, so it depends on the checkpoint that will answer: a question with 50 options is accepted on multilingual input and rejected on English input.

How a question is read: each question is encoded as one sequence of the type and instructions, then every option, then the state, capped at 8,192 tokens. The instructions and options share a budget of head_max_len tokens (192 on english, 256 on multilingual), and each option is cut at 48 tokens. When the options do not fit that budget, every option is silently shortened to (head_max_len - 16) / n tokens, where n is the number of options, and never below 4. The instructions then get what is left of the budget, never less than 8 tokens, and are silently cut to it. The option limit above is the largest count at which all options at 4 tokens each still fit the budget; under it, long options and instructions are still cut without an error. The state gets the tokens that remain after the instructions and options, up to about 8,000. That holds most 32 KB states, but not one that tokenizes densely: a number-heavy JSON state or base64 can pass it. A state that runs past it is cut at the end; usage.truncated_tokens is how you see that it happened and how many tokens went unread. Put what matters first anyway (in a string state; see the note on key order above): the model reads long inputs, but its accuracy falls past about 4,000 tokens (see Limits from the model's authors). A long request, where the number of questions times the tokens per question passes 65,536, runs as several forwards over consecutive groups of its questions: the answers equal the one-forward answers up to numerical noise, and usage is the same. To predict a split, count each question as the whole instructions-and-options allowance (192 tokens on english, 256 on multilingual) plus the state tokens plus 4, capped at 8,192: the server splits when that sum over the questions passes 65,536, whether or not the heads use their full allowance. A request under it runs in one forward, unchanged.

Billing

openrelay/laya costs $0.020 per 1M input tokens. It has no output rate, because it generates nothing.

The state is encoded once per question, so a request with ten questions reads the state about ten times, and bills for it. Asking several questions in one request saves round trips, not tokens. A request with 64 questions over a 6,300-token state bills about 400,000 input tokens, or $0.008 at this rate. Billable tokens per request are bounded by questions times the per-question limit: at most 64 x 8,192 = 524,288 tokens, about $0.0105.

usage.prompt_tokens in the response is the number you are billed on; it is the source of truth, not any estimate you compute from the request size. A request that fails with a non-2xx status is not billed. See Pricing & billing.

Errors

A request the model rejects gets a typed error with a documented code and, in most cases, a param naming the field:

{
  "error": {
    "message": "too many options for one question",
    "type": "invalid_request_error",
    "code": "too_many_options",
    "param": "questions.department.criteria"
  }
}

The codes this family can return, all 400 with type invalid_request_error:

param is a dotted path into your own request, with [n] for array indices. It is present only when that path exists in what you sent, so a missing field (for example a question with no instructions) comes back without a param. It is also omitted when the path is longer than 128 bytes or passes through a question id that contains a dot, a space, or any character other than ASCII letters, digits, _, and -.

Every other error is the gateway's, listed on Errors: 401 for the key, 402 for the balance, 404 for an unknown model, 400 unsupported_api for a model that does not serve decisions, 413 for the body size, 429 when the model's queue is full, and 502 or 503 while the model is restarting or has no ready capacity. Retry 429, 502, and 503 with backoff (after the Retry-After header on a 429); do not retry a 400 unchanged.

Limits from the model's authors

The left column comes from the Laya README, not from our measurements:

Examples

curl https://inference.openrelay.inc/v1/decisions \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openrelay/laya",
    "state": {
      "subject": "Duplicate charge on invoice #4411",
      "body": "I was charged twice. If this is not fixed today I will cancel."
    },
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which department handles this?",
        "criteria": { "billing": "invoices, refunds", "technical": "bugs, outages" }
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent?",
        "criteria": ["not urgent", "soon", "blocking"]
      },
      "churn_risk": {
        "type": "noul",
        "instructions": "Does the user threaten to leave?"
      }
    }
  }'