Provider nodes

This page covers what a provider box needs, how each runtime sets up GPUs, how to confirm a node is healthy, and how nodes behave over their lifetime. To bring your first node online, start with Providers: onboarding is one command. Everything here is handled for you by that command; it's documented so you know what's happening and how to operate the box afterwards.

Hardware requirementsCopy Anchor Link

Both node typesCopy Anchor Link

VM nodes (qemu)Copy Anchor Link

Pod nodes (runc)Copy Anchor Link

GPU setupCopy Anchor Link

VM nodes: passthrough (QEMU + VFIO)Copy Anchor Link

The one-command install configures passthrough for you: you don't hand-edit GRUB or modprobe files. On a GPU box it:

1. Detects your NVIDIA/AMD GPUs with lspci (works even with no driver installed).
2. If they aren't passthrough-ready, enables IOMMU on the kernel command line, binds each GPU (and its IOMMU-group siblings) to vfio-pci, and installs a boot-time service (openrelay-vfio-bind.service) so the binding survives reboots.
3. Asks you to reboot and re-run the exact same command (the installer exits 75). The node is intentionally not enrolled until passthrough actually works: we never advertise a GPU that can't run a VM.

If, after a reboot, the GPUs still aren't bound, enable virtualization + IOMMU in your BIOS/UEFI, then reboot and re-run:

• Intel: enable VT-x and VT-d.
• AMD: enable SVM and IOMMU (set to Enabled, not Auto).

The unit of allocation is the IOMMU group: GPUs that share a group are always passed to a VM together. For full-node (all-GPU) allocations, NVSwitches are passed through too so NVLink works. This is automatic.

Pod nodes: native driver, no VFIOCopy Anchor Link

A pod node does the opposite of the above, and does it automatically:

• The host GPU driver is kept. The VFIO conversion self-skips when a host driver (nvidia or amdgpu) already owns the GPUs, so your driver is never torn down.
• GPUs reach a container per vendor. On NVIDIA the container toolkit is installed and wired into Docker, which is what serves the --gpus flag. AMD has no such toolkit: each pod is granted /dev/kfd plus the render node and card node of its own GPUs, resolved from each GPU's PCI address. Either way the node runs a userns-remapped Docker daemon and the platform bridge.
• No IOMMU groups are required. Cloud and virtualized guests usually expose none; the node then advertises one allocatable unit per GPU, which is exactly how pods consume them.
• The vfio-gpu Nomad plugin is not installed, because it would compete with the container runtime for the same GPUs.

Check node healthCopy Anchor Link

Run the check that matches your node type. Running the wrong one is the most common source of confusing output.

sudo openrelay-node-doctor

Node lifecycleCopy Anchor Link

Two credentials (don't conflate them)Copy Anchor Link

Everything after enrollment runs on the durable identity, so the provisioning token expiring later is by design and harmless.

Reboots & power lossCopy Anchor Link

systemd brings nebula, nomad, and node-agent back at boot; they read their credentials from disk and rejoin automatically. On VM nodes the VFIO bind service re-binds the GPUs before Nomad starts. No action needed.

Automatic updatesCopy Anchor Link

The node-agent keeps itself current via a systemd timer (hourly, with up to 20 minutes of jitter, first run 10 minutes after boot), authenticated with the durable node token. Updates drain in-flight HTTP/SSH first, install atomically, and roll back on failure, so live sessions survive.

• Opt out per node: enroll with NODE_AGENT_AUTOUPDATE=0, or sudo systemctl disable --now node-agent-update.timer.
• Force a check now: sudo systemctl start node-agent-update.service (logs: journalctl -u node-agent-update).

Recycle / re-enrollCopy Anchor Link

Re-running the onboarding one-liner on an existing box is safe and idempotent: the node keeps its identity (same node ID and overlay IP), its token is rotated, no quota is consumed, and nothing referencing the node (VMs, pods) breaks. Use it to repair a node or roll it onto new config.

Take a node offlineCopy Anchor Link

This is the path for a temporary break: maintenance, a reboot, a move. Stop the node's agents:

sudo systemctl stop nomad nebula node-agent

Nomad stops scheduling to it and the data-plane tunnel closes (in-flight work drains first). The node stays enrolled, and VM and pod rentals on it pause while it is away. Start the services again (or reboot) to bring it back, and those rentals resume. A managed Kubernetes worker on the node does not pause: it ends. A node away for more than 24 hours is decommissioned automatically, which ends its rentals; re-run the onboarding one-liner to bring it back after that.

Decommission a nodeCopy Anchor Link

To retire a node permanently, choose Decommission in the node's row menu in the Nodes view of the dashboard, or set its status to removed via the API:

curl -fsSL -X PATCH -H "Authorization: Bearer $VL_KEY" \
  -H "Content-Type: application/json" -d '{"status":"removed"}' \
  https://api.openrelay.inc/v1/orgs/$VL_ORG/provider/nodes/$NODE_ID

This frees the node's workloads and GPU units. A decommission isn't a one-way door: re-running the onboarding one-liner on a removed node auto-recovers it.

Decommission with notice, in the same row menu where it is enabled for your organization, gives renters time first. Pick a deadline from 5 minutes to 7 days out: the node takes no new rentals from the moment you schedule it, rentals already on it keep running until the deadline, and at the deadline they end and the node is removed, the same end state as Decommission. Cancel it from the row any time before the deadline.

Clean offboard (decommission the hardware)Copy Anchor Link

The steps above retire the node in the control plane. Clearing the node-local state is a separate, deeper step you only need when you're decommissioning the hardware for good or moving the box to a different environment and re-onboarding it.

A node's durable identity, Nomad client state, overlay certificates, and config are written to local disk at enrollment. If a box is re-onboarded (or pointed at a different environment) without clearing this, the fresh enrollment collides with the stale identity: the node's Nomad client is rejected and it gets stuck in pending, never coming online. A clean offboard wipes that node-local state so a new enrollment provisions cleanly.

sudo systemctl stop nomad nebula node-agent node-agent-update.timer

sudo rm -rf /opt/nomad/data
sudo rm -rf /etc/nomad /etc/nomad.d
sudo rm -f  /etc/systemd/system/nomad.service.d/intro.conf

sudo rm -rf /etc/nebula

sudo rm -rf /etc/vectorlay/dataplane
sudo rm -f  /etc/vectorlay/node.token /etc/vectorlay/node-agent-update.env \
            /etc/vectorlay/identity.key /etc/vectorlay/vfio-attempts

sudo rm -rf /var/lib/vectorlay/vms/* /var/lib/vectorlay/seed/*

sudo systemctl daemon-reload

Re-onboarding after a clean offboardCopy Anchor Link

Once the node-local state is cleared, re-onboard the box with the standard one-command installer. Any provisioning token that is still within its 24-hour lifetime works; mint a fresh one if the old token has expired or you revoked it:

curl -fsSL "https://api.openrelay.inc/v1/provider/bootstrap?token=<provisioning-token>" | sudo bash

The installer detects the node kind and GPUs on the box, so no extra params are needed. See Providers for how to mint a token and the full set of bootstrap options.

TroubleshootingCopy Anchor Link

See Providers for onboarding and Authentication for API keys.