4× DGX Spark cluster#
Note
Already SSH’d in?
After login, the cluster banner points to /etc/enverge/cluster.md on your instance: which node is on which rail, worker SSH, and a sample torchrun command with this VM’s addresses filled in (you still bring your own script). This page is the generic reference, that file is instance-specific.
A 4× DGX Spark instance is a four-node GPU cluster behind a single SSH endpoint. You land on node0 (the head). Three more Sparks (node1 … node3, the workers) are attached over a switchless RoCE ring: each machine has two ConnectX-7 ports and is cabled to its two neighbours, with no network switch. You do not SSH to the workers to work, you launch a distributed job from node0 and it uses every GPU.
Warning
nvidia-smi shows one GPU
nvidia-smi on node0 reports one GB10 (~128 GB unified). The other three GPUs live on the workers, reachable only through a sharded multi-node job (tensor or pipeline parallel, FSDP, DeepSpeed, vLLM multi-node, Ray). A plain python train.py uses only node0 and leaves the workers idle.
Memory model#
The tier advertises 512 GB unified memory, which is aggregate across four 128 GB pools joined by the fabric, not one flat address space. No single process or tensor can exceed ~128 GB. Use model parallelism or sharding to spread work across all four nodes.
How the ring is wired#
Four point-to-point QSFP cables form a ring. Each cable is its own subnet, shared by the two machines at its ends. Opposite nodes have no direct cable: IP traffic and routes on every node forward through a neighbour, so every rail IP is still reachable from every node. For bulk NCCL collectives between machines with no cable between them, prefer a switchless-aware NCCL that keeps traffic on the neighbour links (for example switchless-nccl). Stock NCCL is fine for bootstrap and small messages over the forwarded path.
Pre-configured on your instance#
NCCL over RoCE, this node’s ConnectX-7 rails are wired for RDMA (
/etc/nccl.conf). NCCL auto-discovers the fabric, no manualNCCL_IB_*exports needed.VLLM_HOST_IP, preset to this node’s rail IP (from
/etc/profile.d/enverge-cluster.sh) so vLLM’s multi-node control plane advertises the correct address instead of the default-route interface.Worker SSH, passwordless SSH from this container to every worker over the fabric is already set up for launchers that spawn remote ranks. Check
/etc/enverge/cluster.mdon the instance for this VM’s rail IPs and the three worker addresses.
Running a multi-node job#
Launch rank 0 here and ranks 1…3 on the workers over the fabric. Any NCCL-based stack works once the workload is sharded. The shape is:
torchrun --nnodes=4 --nproc_per_node=1 --node_rank=0 \
--rdzv_endpoint=<node0-rail-ip>:29400 your_job.py
your_job.py is your distributed script, nothing is pre-installed to run for you. Set --node_rank=1 (and 2, 3) on the workers, or let Ray or mpirun spawn the remote ranks via the pre-configured fabric SSH. Replace <node0-rail-ip> with this instance’s rendezvous address (see /etc/enverge/cluster.md for the filled-in template). A launcher that has not seen the workers’ SSH host keys yet needs -o StrictHostKeyChecking=accept-new, or SSH to each worker once by hand first.
Unified memory on GB10#
Warning
Cap GPU memory utilization
On GB10, “GPU memory” is unified with system RAM. Settings that grab all GPU memory, for example vLLM gpu_memory_utilization (default 0.9), can reach for ~110 GB and get killed by the host memory guard. Use a conservative cap (~0.5, never 0.9) on memory-heavy jobs.