Managed Kubernetes

A managed Kubernetes cluster has a control plane we run and nodes that are whole OpenRelay hosts, either GPU or CPU, in one data center. You create and size it through the Kubernetes API or the orl kube commands, and use it with kubectl.

What a cluster isCopy Anchor Link

• Kubernetes v1.37.1. It is the only version offered, and a cluster is not upgraded in place.
• One data center. A cluster's nodes all run in the location you create it in (locationId, from orl catalog location list or GET /v1/locations).
• Node pools of whole hosts. Every node in a pool is the same SKU: one GPU model (gpuModelId, from orl catalog gpu-model list or GET /v1/gpu-models) or one whole-host CPU SKU (cpuSku). A GPU node is a whole host of that model with every GPU passed through to it. A pool holds 1 to 100 nodes; mix SKUs by adding several pools.
• Credentials from your API key. The API exchanges your key for a cluster token: a bearer token only that cluster accepts, carrying cluster-admin, which expires after one hour. There are no client certificates to manage.
• Fixed networks. Pods get addresses from 10.244.0.0/14 and services from 10.96.0.0/16 in every cluster.

Reading clusters, pools and nodes needs Kubernetes: Read; getting a kubeconfig or token needs Kubernetes: Connect; creating, scaling or deleting needs Kubernetes: Manage (Permissions). A legacy key needs clusters:read to read and clusters:write for the rest.

1. Create a clusterCopy Anchor Link

orl kube cluster create --name training --location-id <locationId>

The same request as an API call is POST /v1/kubernetes/clusters:

curl -X POST https://api.openrelay.inc/v1/kubernetes/clusters \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "training", "locationId": "<locationId>"}'

It answers 202 with the cluster's id and status: creating. Poll the cluster until its status is ready:

orl kube cluster get <cluster-id>

A name is lowercase letters, digits and hyphens, starts with a letter, does not end with a hyphen, and is at most 63 characters. It is unique within your organization until the cluster's deletion completes.

You can also pass up to 10 node pools in the create request (nodePools, each shaped like the node pool request below). They are added once the control plane is ready.

2. Add a node poolCopy Anchor Link

Once the cluster is ready, add nodes:

orl kube node-pool create <cluster-id> \
  --name gpu \
  --count 2 \
  --sku '{"gpuModelId": "<gpuModelId>"}'

As an API call, POST /v1/kubernetes/clusters/{id}/node-pools takes {"name": "gpu", "sku": {"gpuModelId": "<gpuModelId>"}, "count": 2}. For a CPU pool, the SKU is {"cpuSku": "<cpuSku>"}.

The pool reaches its count as hosts are claimed and their nodes join; until then its readyCount is lower. Watch the nodes:

orl kube node list <cluster-id>

A node is registered while its host boots, then joined once it is in the cluster, where kubernetesStatus reports its Ready condition.

3. Use kubectlCopy Anchor Link

orl kube kubeconfig writes a kubeconfig for the cluster. It holds no credential: kubectl runs orl kube token to get a cluster token, and a new one whenever the last one expires, so orl must be installed where kubectl runs.

orl auth login   # once, if you have not stored a key yet
orl kube kubeconfig --cluster <cluster-id> --output ~/.kube/openrelay
export KUBECONFIG=~/.kube/openrelay
kubectl get nodes

Without the CLI, GET /v1/kubernetes/clusters/{id}/kubeconfig returns a kubeconfig with a cluster token embedded, which stops working when the token expires after one hour, and POST /v1/kubernetes/clusters/{id}/token returns a token as a client.authentication.k8s.io/v1 ExecCredential for your own credential plugin.

4. Run on a GPUCopy Anchor Link

Request GPUs with the nvidia.com/gpu resource. Each GPU on a node is one, and a container that requests none sees no GPU:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-check
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvcr.io/nvidia/cuda:12.8.1-base-ubuntu24.04
      command: ["nvidia-smi", "-L"]
      resources:
        limits:
          nvidia.com/gpu: 1

kubectl apply -f gpu-check.yaml
kubectl logs gpu-check

Scale and deleteCopy Anchor Link

orl kube node-pool scale <cluster-id> <pool-id> --count 4   # 1 to 100
orl kube node delete <cluster-id> <node-id>                 # drains it first
orl kube node-pool delete <cluster-id> <pool-id>
orl kube cluster delete <cluster-id>

• Scaling down drains the nodes it removes before removing them.
• Deleting a node drains it first unless you pass --drain=false (?drain=false on the API). The pool's count drops by one, so the node is not replaced. A pool's last node cannot be deleted on its own: delete the pool.
• Draining cordons the node and evicts every pod on it except a DaemonSet's and any pod that tolerates the cordon (node.kubernetes.io/unschedulable). The drain does not wait on those: they get no eviction, so no disruption budget is consulted, and they stop when the node is removed.
• Taints. The managed GPU operator and MPI operator are scheduled only onto nodes with no NoSchedule or NoExecute taint of yours. Cluster DNS, the Cilium operator and the local-path provisioner run through your NoSchedule taints but leave a node with a NoExecute taint of yours. Keep at least one node with neither so they always have a place to run.
• Deleting a pool drains and removes every node in it.
• Deleting a cluster removes its pools and every node, and is final. orl kube cluster get answers 404 once it completes. Repeating the delete finishes one that could not remove everything.

When a request is refusedCopy Anchor Link