Get the org's on-demand inference series

Requests, error rate, latency and time-to-first-token percentiles, and tokens for the org's online on-demand inference over 1h, 6h, 24h, or 7d. Every series shares one time grid, and `models` lists the models with traffic in the range. Without `model` every series is the org total across its on-demand models. Batch and async calls are not counted. Every series also carries a whole-range figure, and the per-point counts add up to it. Data appears within a minute of the org's first on-demand request. These are display metrics, not billing: spend is in the daily usage endpoint.

Required permission

Requires the usage.read permission. See Permissions for the roles and API key permissions that include it.

Command line

The orl CLI wraps this endpoint:

orl usage inference-series get

GET

/v1/orgs/{orgId}/inference/series

GET

/v1/orgs/{orgId}/inference/series

Authorization

apiKey

headerAuthorizationBearer <token>

OpenRelay API key. Send it as Authorization: Bearer or_… (legacy vl_… keys still work).

Path Parameters

orgId*string

Query Parameters

range?string

the range to read; 1h when absent

Default"1h"

Value in"1h""6h""24h""7d"

model?string

a public model id to filter to

Length1 <= length <= 256

Response Body

Series

application/json

response

model?string

the public model id every series is filtered to; absent for the org total across models

models*array<string>

the public model ids with on-demand traffic in the range, sorted

range*string

Value in"1h""6h""24h""7d"

start*string

the first grid point

Formatdate-time

end*string

the last grid point

Formatdate-time

stepSeconds*integer

pointCount*integer

the length of every series' values

metrics*

Every chart, always all six. requests: success and error per second. errors: client_error, rate_limited and server_error, each as a share of all requests. latency (request start to last byte) and ttft (streaming requests only): p50, p90, p95 and p99 in seconds. tokens: per second by kind (input, cached_input, cache_write, output), only the kinds metered in the range. concurrency (always sent; optional for clients of older answers): requests in flight, averaged over each point's step, as series in_flight. A request is counted when it finishes, its whole duration in that step, so a point can read above the endpoint's capacity when long requests finish together; its window, the average over the whole range, is the steady figure.

Example Request

curl -X GET 'https://api.openrelay.inc/v1/orgs/string/inference/series'

Example Request

curl -X GET 'https://api.openrelay.inc/v1/orgs/string/inference/series'

Example ResponseSchema

{
  "model": "string",
  "models": [
    "string"
  ],
  "range": "1h",
  "start": "2019-08-24T14:15:22Z",
  "end": "2019-08-24T14:15:22Z",
  "stepSeconds": 0,
  "pointCount": 0,
  "metrics": {
    "requests": {
      "unit": "requests_per_second",
      "series": [
        {
          "key": "string",
          "values": [
            0.1
          ],
          "window": 0.1
        }
      ]
    },
    "errors": {
      "unit": "requests_per_second",
      "series": [
        {
          "key": "string",
          "values": [
            0.1
          ],
          "window": 0.1
        }
      ]
    },
    "latency": {
      "unit": "requests_per_second",
      "series": [
        {
          "key": "string",
          "values": [
            0.1
          ],
          "window": 0.1
        }
      ]
    },
    "ttft": {
      "unit": "requests_per_second",
      "series": [
        {
          "key": "string",
          "values": [
            0.1
          ],
          "window": 0.1
        }
      ]
    },
    "tokens": {
      "unit": "requests_per_second",
      "series": [
        {
          "key": "string",
          "values": [
            0.1
          ],
          "window": 0.1
        }
      ]
    },
    "concurrency": {
      "unit": "requests_per_second",
      "series": [
        {
          "key": "string",
          "values": [
            0.1
          ],
          "window": 0.1
        }
      ]
    }
  }
}

{  "model": "string",  "models": [    "string"  ],  "range": "1h",  "start": "2019-08-24T14:15:22Z",  "end": "2019-08-24T14:15:22Z",  "stepSeconds": 0,  "pointCount": 0,  "metrics": {    "requests": {      "unit": "requests_per_second",      "series": [        {          "key": "string",          "values": [            0.1          ],          "window": 0.1        }      ]    },    "errors": {      "unit": "requests_per_second",      "series": [        {          "key": "string",          "values": [            0.1          ],          "window": 0.1        }      ]    },    "latency": {      "unit": "requests_per_second",      "series": [        {          "key": "string",          "values": [            0.1          ],          "window": 0.1        }      ]    },    "ttft": {      "unit": "requests_per_second",      "series": [        {          "key": "string",          "values": [            0.1          ],          "window": 0.1        }      ]    },    "tokens": {      "unit": "requests_per_second",      "series": [        {          "key": "string",          "values": [            0.1          ],          "window": 0.1        }      ]    },    "concurrency": {      "unit": "requests_per_second",      "series": [        {          "key": "string",          "values": [            0.1          ],          "window": 0.1        }      ]    }  }}

Get the org's on-demand inference series

Required permissionCopy Anchor Link

Command lineCopy Anchor Link

AuthorizationCopy Anchor Link

Path ParametersCopy Anchor Link