Provider nodes

This page covers what a provider box needs, how each runtime sets up GPUs, how to confirm a node is healthy, and how nodes behave over their lifetime. To bring your first node online, start with Providers — onboarding is one command. Everything here is handled for you by that command; it's documented so you know what's happening and how to operate the box afterwards.

Hardware requirements

Both node types

VM nodes (qemu)

Pod nodes (runc)

GPU setup

VM nodes: passthrough (QEMU + VFIO)

The one-command install configures passthrough for you — you don't hand-edit GRUB or modprobe files. On a GPU box it:

1. Detects your NVIDIA/AMD GPUs with lspci (works even with no driver installed).
2. If they aren't passthrough-ready, enables IOMMU on the kernel command line, binds each GPU (and its IOMMU-group siblings) to vfio-pci, and installs a boot-time service (vectorlay-vfio-bind.service) so the binding survives reboots.
3. Asks you to reboot and re-run the exact same command (the installer exits 75). The node is intentionally not enrolled until passthrough actually works — we never advertise a GPU that can't run a VM.

If, after a reboot, the GPUs still aren't bound, enable virtualization + IOMMU in your BIOS/UEFI, then reboot and re-run:

• Intel — enable VT-x and VT-d.
• AMD — enable SVM and IOMMU (set to Enabled, not Auto).

The unit of allocation is the IOMMU group: GPUs that share a group are always passed to a VM together. For full-node (all-GPU) allocations, NVSwitches are passed through too so NVLink works — this is automatic.

Pod nodes: native driver, no VFIO

A pod node does the opposite of the above, and does it automatically:

• The host NVIDIA driver is kept. The VFIO conversion self-skips when nvidia-smi is present, so your driver is never torn down.
• The NVIDIA container toolkit is installed and wired into Docker so --gpus works, along with a userns-remapped Docker daemon and the platform bridge.
• No IOMMU groups are required. Cloud and virtualized guests usually expose none; the node then advertises one allocatable unit per GPU, which is exactly how pods consume them.
• The vfio-gpu Nomad plugin is not installed, because it would compete with the container runtime for the same GPUs.

Check node health

Run the check that matches your node type. Running the wrong one is the most common source of confusing output.

sudo vectorlay-node-doctor

Node lifecycle

Two credentials (don't conflate them)

Everything after enrollment runs on the durable identity — the provisioning token expiring later is by design and harmless.

Reboots & power loss

systemd brings nebula, nomad, and node-agent back at boot; they read their credentials from disk and rejoin automatically. On VM nodes the VFIO bind service re-binds the GPUs before Nomad starts. No action needed.

Automatic updates

The node-agent keeps itself current via a systemd timer (hourly, with up to 20 minutes of jitter, first run 10 minutes after boot), authenticated with the durable node token. Updates drain in-flight HTTP/SSH first, install atomically, and roll back on failure — live sessions survive.

• Opt out per node: enroll with NODE_AGENT_AUTOUPDATE=0, or sudo systemctl disable --now node-agent-update.timer.
• Force a check now: sudo systemctl start node-agent-update.service (logs: journalctl -u node-agent-update).

Recycle / re-enroll

Re-running the onboarding one-liner on an existing box is safe and idempotent: the node keeps its identity (same node ID and overlay IP), its token is rotated, no quota is consumed, and nothing referencing the node (clusters / VMs) breaks. Use it to repair a node or roll it onto new config.

Take a node offline

To temporarily remove a node from the network, stop its agents:

sudo systemctl stop nomad nebula node-agent

Nomad stops scheduling to it and the data-plane tunnel closes (in-flight work drains first). Start the services again — or reboot — to bring it back.

Decommission a node

To retire a node permanently, set its status to removed from the Nodes view in the dashboard, or via the API:

curl -fsSL -X PATCH -H "Authorization: Bearer $VL_KEY" \
  -H "Content-Type: application/json" -d '{"status":"removed"}' \
  https://api.openrelay.inc/v1/orgs/$VL_ORG/provider/nodes/$NODE_ID

This frees the node's workloads and GPU units. A decommission isn't a one-way door: re-running the onboarding one-liner on a removed node auto-recovers it.

Clean offboard (decommission the hardware)

The steps above retire the node in the control plane. Clearing the node-local state is a separate, deeper step you only need when you're decommissioning the hardware for good or moving the box to a different environment and re-onboarding it.

A node's durable identity, Nomad client state, overlay certificates, and config are written to local disk at enrollment. If a box is re-onboarded — or pointed at a different environment — without clearing this, the fresh enrollment collides with the stale identity: the node's Nomad client is rejected and it gets stuck in pending, never coming online. A clean offboard wipes that node-local state so a new enrollment provisions cleanly.

sudo systemctl stop nomad nebula node-agent node-agent-update.timer

sudo rm -rf /opt/nomad/data
sudo rm -rf /etc/nomad /etc/nomad.d
sudo rm -f  /etc/systemd/system/nomad.service.d/intro.conf

sudo rm -rf /etc/nebula

sudo rm -rf /etc/vectorlay/dataplane
sudo rm -f  /etc/vectorlay/node.token /etc/vectorlay/node-agent-update.env \
            /etc/vectorlay/identity.key /etc/vectorlay/vfio-attempts

sudo rm -rf /var/lib/vectorlay/vms/* /var/lib/vectorlay/seed/*

sudo systemctl daemon-reload

Re-onboarding after a clean offboard

Once the node-local state is cleared, re-onboard the box with the standard one-command installer. Any provisioning token that is still within its 24-hour lifetime works; mint a fresh one if the old token has expired or you revoked it:

curl -fsSL "https://api.openrelay.inc/v1/provider/bootstrap?token=<provisioning-token>" | sudo bash

The installer detects the node kind and GPUs on the box, so no extra params are needed. See Providers for how to mint a token and the full set of bootstrap options.

Troubleshooting

See Providers for onboarding and Authentication for API keys.