Provider nodes

This page covers what a provider box needs, how GPU passthrough is set up, how to confirm a node is healthy, and how nodes behave over their lifetime. To bring your first node online, start with Providers — onboarding is one command. Everything here is handled for you by that command; it's documented so you know what's happening and how to operate the box afterwards.

Hardware requirements

NVIDIA GPU passthrough

The one-command install configures passthrough for you — you don't hand-edit GRUB or modprobe files. On a GPU box it:

1. Detects your NVIDIA/AMD GPUs with lspci (works even with no driver installed).
2. If a host NVIDIA driver stack is preinstalled (common on datacenter images — driver + fabricmanager + DCGM, e.g. HGX H100 SXM boxes), the installer converts the box automatically: it stops and disables the NVIDIA host services, blacklists the driver stack, and unbinds the GPUs. When IOMMU is already active this happens live with no reboot; otherwise the standard reboot + re-run finishes it. (Keep the host driver deliberately with NODE_KEEP_HOST_DRIVER=1 — the node then sells no VM GPUs.)
3. If they aren't passthrough-ready, enables IOMMU on the kernel command line, binds each GPU (and its IOMMU-group siblings) to vfio-pci, and installs a boot-time service (vectorlay-vfio-bind.service) so the binding survives reboots.
4. Asks you to reboot and re-run the exact same command. The node is intentionally not enrolled until passthrough actually works — we never advertise a GPU that can't run a VM.

If, after a reboot, the GPUs still aren't bound, enable virtualization + IOMMU in your BIOS/UEFI, then reboot and re-run:

• Intel — enable VT-x and VT-d.
• AMD — enable SVM and IOMMU (set to Enabled, not Auto).

Multi-GPU nodes

The unit of allocation is the IOMMU group: GPUs that share a group are always passed to a VM together. The installer binds every NVIDIA/AMD GPU on the host to vfio-pci. For full-node (all-GPU) allocations, NVSwitches are passed through too so NVLink works — this is automatic.

On SXM/NVLink platforms (HGX H100/B200) the NVSwitch fabric devices are bound to vfio-pci alongside the GPUs and recorded at enrollment. A VM that rents the whole node gets the NVSwitches passed through and NVLink enabled in-guest (fabricmanager is set up automatically); partial allocations run with NVLink disabled — the fabric is shared hardware and is never split between tenants.

GPU runtime modes

• Passthrough (QEMU + VFIO) — the default for GPU boxes with no host driver. The VM gets bare-metal GPU performance.
• Container GPU (gVisor + nvproxy) — pass gpu=1 to the bootstrap. GPUs stay bound to the host NVIDIA driver and are shared via nvproxy; no IOMMU/VFIO needed.

Check node health

After install — or any time — run the built-in doctor:

sudo vectorlay-node-doctor

It prints PASS / FAIL per check and a final verdict:

• Agents & tunnel — nebula, nomad, and node-agent running; auto-update timer enabled.
• Virtualization — /dev/kvm present.
• GPU passthrough — IOMMU active and per-IOMMU-group VFIO readiness, ending with N/M GPU unit(s) ready to rent.
• Disk — enough free space for VM base images.

A green doctor means the control plane will advertise the node's GPU units.

• If a unit shows WEDGED, a GPU fell off the bus (the D3cold reset bug) — power-cycle the host (a soft reboot may not clear it).
• If a unit isn't bound to vfio-pci, run sudo systemctl restart vectorlay-vfio-bind or re-run the bootstrap.

Node lifecycle

Two credentials (don't conflate them)

Everything after enrollment runs on the durable identity — the provisioning token expiring an hour later is by design and harmless.

Reboots & power loss

systemd brings nebula, nomad, and node-agent back at boot; they read their credentials from disk and rejoin automatically. On GPU nodes the VFIO bind service re-binds the GPUs before Nomad starts. No action needed.

Automatic updates

The node-agent keeps itself current via a jittered systemd timer (~hourly + up to 20 min), authenticated with the durable node token. Updates drain in-flight HTTP/SSH first, install atomically, and roll back on failure — live sessions survive.

• Opt out per node: enroll with NODE_AGENT_AUTOUPDATE=0, or sudo systemctl disable --now node-agent-update.timer.
• Force a check now: sudo systemctl start node-agent-update.service (logs: journalctl -u node-agent-update).

Recycle / re-enroll

Re-running the onboarding one-liner on an existing box is safe and idempotent: the node keeps its identity (same node ID and overlay IP), its token is rotated, no quota is consumed, and nothing referencing the node (clusters / VMs) breaks. Use it to repair a node or roll it onto new config.

Take a node offline

To temporarily remove a node from the network, stop its agents:

sudo systemctl stop nomad nebula node-agent

Nomad stops scheduling to it and the data-plane tunnel closes (in-flight work drains first). Start the services again — or reboot — to bring it back.

Decommission a node

To retire a node permanently, open the Nodes view in the dashboard, click Decommission on the node's row, and confirm. That is the whole flow, and it needs no credentials beyond your dashboard login.

Both terminal paths below need an org API key (vl_…) with the clusters:write scope, created on the Account → API Keys page. The provisioning token you installed the node with (vtk_…) is a different credential and is rejected here.

With the CLI (install), log in once and every later command reuses the key. A key is bound to one org, so orl reads your org from it and you never pass an org id:

orl auth login    # one time, paste your vl_… key
orl provider node update $NODE_ID --status removed

Calling the API directly is the only path that needs your org id in the URL (shown under your org name in the dashboard sidebar, with a copy button):

curl -sS -X PATCH -H "Authorization: Bearer $VL_KEY" \
  -H "Content-Type: application/json" -d '{"status":"removed"}' \
  https://api.openrelay.inc/v1/orgs/$VL_ORG/provider/nodes/$NODE_ID

Any of the three frees the node's workloads and GPU units. A decommission isn't a one-way door: re-running the onboarding one-liner on a removed node auto-recovers it.

Clean offboard (decommission the hardware)

The steps above retire the node in the control plane. Clearing the node-local state is a separate, deeper step you only need when you're decommissioning the hardware for good or moving the box to a different environment and re-onboarding it.

A node's durable identity, Nomad client state, overlay certificates, and config are written to local disk at enrollment. If a box is re-onboarded — or pointed at a different environment — without clearing this, the fresh enrollment collides with the stale identity: the node's Nomad client is rejected and it gets stuck in pending, never coming online. A clean offboard wipes that node-local state so a new enrollment provisions cleanly.

sudo systemctl stop nomad nebula node-agent node-agent-update.timer

sudo rm -rf /opt/nomad/data
sudo rm -rf /etc/nomad /etc/nomad.d
sudo rm -f  /etc/systemd/system/nomad.service.d/intro.conf

sudo rm -rf /etc/nebula

sudo rm -rf /etc/vectorlay/dataplane
sudo rm -f  /etc/vectorlay/node.token /etc/vectorlay/node-agent-update.env /etc/vectorlay/node.env

sudo rm -rf /var/lib/vectorlay/vms/* /var/lib/vectorlay/seed/*

sudo systemctl daemon-reload

Re-onboarding after a clean offboard

Once the node-local state is cleared, re-onboard the box with the standard one-command installer using a fresh provisioning token — the old token is single-use and short-lived:

curl -fsSL "https://api.openrelay.inc/v1/provider/bootstrap?token=<provisioning-token>&nodeType=persistent" | sudo bash

Add &gpu=1 for GPU nodes. See Providers for how to mint a token and the full set of bootstrap options.

Troubleshooting

See Providers for onboarding and Authentication for API keys.