Pods and persistent storage

A pod runs your container image on one or more whole GPUs of a single node, with SSH access and an HTTPS endpoint. Every pod comes with a persistent volume mounted read-write at /workspace. That volume is what makes a pod safe to stop and start: your data lives there, not in the container filesystem.

What persistsCopy Anchor Link

Put checkpoints, datasets, notebooks and anything you cannot re-download under /workspace. Package installs, files written elsewhere and running processes are gone whenever the container is recreated.

Volume sizeCopy Anchor Link

Pick the size when you create the pod (volumeGb): the default is 50 GB, the range is 10 to 1024 GB. The size is fixed for the pod's life; there is no resize. The volume lives on the pod's node, is not billed separately and adds nothing to the hourly rate.

The pod page in the dashboard shows how much of the volume is in use. The figure comes from an hourly report by the node, so it can lag by up to an hour.

Creating a podCopy Anchor Link

In the dashboard, Pods > New pod has a field for the volume. From the CLI or the API, pass --volume-gb / volumeGb (omit it for 50 GB):

orl pods create \
  --image ghcr.io/you/trainer:latest \
  --gpu-model-id <gpuModelId from orl pods capacity list> \
  --gpu-count 1 \
  --volume-gb 200 \
  --ssh-key-ids <sshKeyId from orl ssh-keys list> \
  --name trainer

The same request as an API call is POST /v1/pods. GET /v1/pods/capacity lists the GPU models with free capacity and their rates.

When a create is refusedCopy Anchor Link

A create that is refused says why in its status and code:

A 503 is a capacity refusal: the request is valid and can succeed once capacity frees up, but sending it again in a tight loop will not make room. orl pods create exits 7 on a 503 and 6 on a 409.

AMD podsCopy Anchor Link

AMD Instinct pod hosts run ROCm on the amdgpu driver. Volume, SSH, lifecycle and billing work exactly as above; what changes is the image.

The image has to match the GPU vendor. A CUDA image on an AMD GPU, or a ROCm image on an NVIDIA GPU, starts a pod whose container cannot see its GPU: torch.cuda.is_available() comes back false and you are billed for the pod anyway. The dashboard blocks the mismatch before you deploy. The API and the CLI do not, so check the image yourself when you create a pod with orl or POST /v1/pods.

Images that work on AMD:

A pod gets whole GPUs. Each one is granted as /dev/kfd plus that GPU's render node, so rocm-smi inside the container lists exactly the pod's GPUs and nothing else. Run it first thing after you SSH in (nvidia-smi on an NVIDIA pod) to confirm the container sees what you paid for.

LifecycleCopy Anchor Link

A pod moves through the statuses below. Only running is billed.

Stop and startCopy Anchor Link

Stop (POST /v1/vms/{id}/stop, orl vms stop) ends the container, releases the GPU and keeps the volume. You are not billed while the pod is stopped.

Start (POST /v1/vms/{id}/restart, orl vms restart on a stopped pod) brings it back on the same node, because that is where the volume is. The GPUs it needs must be free on that node; if another workload has taken them the start is refused with 503 POD_NODE_BUSY, and you can try again later or delete the pod and create a new one elsewhere. An InfiniBand-enabled pod whose fabric has no free address is refused the same way, with 503 INSUFFICIENT_IP_CAPACITY (capacity refusals). A start while the node is offline is refused with 409 POD_NODE_OFFLINE, and a start of an InfiniBand-enabled pod on a node whose agent is still being updated for it with 409 POD_NODE_NO_INFINIBAND. A start of an AMD GPU pod on a node whose agent is below the level that grants AMD GPUs is refused with 409 POD_NODE_NO_AMD_GRANT until the provider updates the agent.

Restart while runningCopy Anchor Link

The same restart call on a running pod recreates the container from the image (a fresh filesystem outside /workspace) and keeps the volume. Billing continues through the restart.

Crashes and node lossCopy Anchor Link

If the container exits or is killed, the pod is recreated automatically on the same node with the volume intact, and the pod stays running.

If the node stops reporting to us, the pod becomes interrupted after a few minutes of grace: billing stops, the volume and the GPU reservation stay with the node, and the pod returns to running by itself once the node is back. You can also stop or delete an interrupted pod.

DeleteCopy Anchor Link

Delete (POST /v1/vms/{id}/terminate, orl vms terminate) removes the pod and its volume. There is no undo and no snapshot: copy anything you need out of /workspace first.

Retention: when a volume is deleted without you askingCopy Anchor Link

Two situations delete a pod and its volume automatically. Both send an email first.

• Node never came back. An interrupted pod whose node has not returned within 7 days is deleted. If the node is removed from the platform before that, the pod is deleted at that point, because the volume was on the box.
• Insufficient balance. When your available balance drops below $0.50, running pods are stopped and kept with their volumes. If the account is not funded again within 7 days of that stop, those pods and their volumes are deleted. Top up before then and start them again.

A pod you stopped yourself is kept, with its volume, until you delete it.

BillingCopy Anchor Link

A pod bills the GPU rate shown at create for every second it is running, as one usage record per running stint (create to stop, start to interrupt, and so on). Stopped and interrupted time is free. The volume has no charge of its own. See VM billing for how rates, balance and auto-stop work; the same rules apply to pods.