What Counts as “Inference”? A Map of Workloads — and What Fits on a DGX Spark
TL;DR: Inference is not chat. It spans embeddings, agents, speech, vision, batch scoring, and diffusion — each with different bottlenecks. DGX Spark wins on local dev, agent prototyping, RAG pipelines, and offline batch where 128 GB unified memory matters more than datacenter bandwidth. It loses on high-concurrency API serving and rack-scale training.
Monday standup. Someone says the product needs “inference.” The PM means a chat API. The ML engineer means embeddings for search. Security asks whether Whisper runs on the same box as the 70B model. Nobody names concurrency, modality, or whether they care about first-token latency versus total tokens per hour.
That meeting is why this post exists.
In this post
- Inference is not one thing
- A taxonomy beyond chat
- Prefill vs decode: the physics on Spark
- Interactive vs offline: topology matters
- What Spark is built for
- Fit guide: excellent, moderate, and weak
- Modality notes
- When to cluster two Sparks
- Frequently asked questions
- Conclusion
- Glossary
Inference is not one thing
In ML jargon, inference is any forward pass: run a trained model on new data and read the output. Training and fine-tuning are out of scope here — one sentence of contrast is enough. Inference is what you ship.
The word narrowed anyway. Chat UIs and OpenAI-compatible APIs made “inference” sound like streaming tokens into a browser. That is one sub-type. The same GB10 Grace Blackwell chip on a DGX Spark might instead be embedding ten million chunks overnight, running Whisper on a folder of calls, or serving a tool-calling agent that never prints a chat bubble. Different phases, different bottlenecks, different fit on 128 GB of unified memory and ~273 GB/s of LPDDR5x.
A taxonomy beyond chat
| Family | Sub-types (beyond chat) | Typical stack |
|---|---|---|
| Text LLM | Code assist, tool agents, structured JSON | vLLM, SGLang, Ollama, TensorRT-LLM |
| RAG | Embeddings, reranking, chunk retrieval, document QA | TEI, sentence-transformers, cross-encoders |
| Batch text | Summarization, eval harnesses, synthetic data | Same engines, batch APIs |
| Speech | ASR, TTS, voice agents | Whisper, Piper, CosyVoice |
| Vision | OCR, detection, segmentation, video analytics | TensorRT, Triton, OpenCV + DNN |
| Multimodal | VLM QA, document understanding | LLaVA-class, Nemotron-VLM |
| Generative media | Image/video diffusion | SD/Flux, ComfyUI |
| Edge frameworks | Robotics perception, Metropolis pipelines | NVIDIA Isaac, Metropolis |
Agents are the messy case: one “agent” often chains LLM decode, embedding calls, vision on uploads, and speech on the side. Map the chain before you map the hardware.
Prefill vs decode: the physics on Spark
LLM inference splits into two phases. Prefill processes the whole prompt in parallel — attention over every input token at once. It is compute-heavy; GB10’s fifth-generation Tensor Cores (with FP4 support) do real work here. Decode generates one token at a time, each step reading the full KV cache (key-value cache — stored attention state). Decode is memory-bandwidth-heavy.
On Spark, that bandwidth ceiling is 273 GB/s shared across CPU and GPU. LMSYS’s Spark review measured the split clearly: Llama 3.1 8B in SGLang at batch 32 hit roughly 7,950 tokens/s prefill and 368 tokens/s decode on Spark, while Llama 3.1 70B (FP8) loaded and ran at ~803 prefill / 2.7 decode. The big model fits; decode crawls. GPT-OSS 20B in Ollama reported ~2,053 prefill / 49.7 decode on the same box.
Prefill-heavy jobs — long context ingestion, batch scoring with fat prompts — tolerate Spark better than “keep 200 users under 50 ms inter-token latency.”
Callout: UMA = one pool. CPU and GPU share 128 GB of coherent LPDDR5x; there is no discrete VRAM carve-out. A 70B weights load plus KV cache competes with SSH, systemd, and your embedding job. Watch MemAvailable, not nvidia-smi memory fields that often read N/A on GB10. Same lesson as bare metal vs container on Spark.
Interactive vs offline: topology matters
Interactive serving cares about per-request speed: time to first token (TTFT), inter-token latency (ITL), concurrency under a service level objective (SLO). Offline batch cares about aggregate throughput — total tokens per hour, no per-user floor.
Those goals pick different cluster shapes. Tensor parallelism (TP) shards one model across nodes so each user’s request stays on a faster logical GPU. Independent replicas (N single-node servers) multiply total throughput but do not make any one request faster.
Cluster topology is not a v2 detail. Openzeka’s GB10 cluster study on Qwen3.6-27B NVFP4 over multi-node vLLM found tensor parallelism across ConnectX-7 Ethernet can lift per-user decode when you need an interactive SLO, while N independent single-node replicas maximize total tokens per hour for offline jobs — same fleet, different product. Treat that report as one lab’s numbers on one model stack, not gospel beside NVIDIA or LMSYS — but the TP-versus-replica trade-off is real on GB10.
What Spark is built for
NVIDIA positions DGX Spark as a desktop Grace Blackwell box for prototyping, deployment, and fine-tuning before you move work to DGX Cloud or other accelerated datacenters. The hardware guide lists models up to 200 billion parameters on one unit, or 405 billion in a dual-Spark configuration — capacity marketing for FP4-class quantization, not a promise of datacenter tok/s.
LMSYS’s conclusion matches the lab charts: Spark loads Llama 70B and GPT-OSS 120B, but those runs are for experimentation, not production serving. Where Spark looked strongest was smaller open models with batching — Llama 3.1 8B scaling decode from ~20 tok/s at batch 1 to ~368 tok/s at batch 32 in SGLang. Speculative decoding (EAGLE3) bought up to ~2× end-to-end throughput on some models in their tests, which matters when bandwidth is the cap.
Spark also has no MIG and no licensed vGPU on GB10. One active GPU tenant gets the whole chip; sharing means queueing or time-slicing, not hardware partitions.
Batch text workloads — overnight summarization, synthetic data generation, regression evals against a golden set — rarely need a per-request latency floor. They need predictable total throughput and a queue that does not wedge the box. Spark is aligned with that shape when you treat replicas as throughput multipliers rather than trying to speed up one straggler request with tensor parallel.
Fit guide: excellent, moderate, and weak
| Workload | Spark fit | Why | Better elsewhere when |
|---|---|---|---|
| Local dev / prompt engineering (7–20B) | Excellent | ~20–50 tok/s decode is a usable iteration loop; offline, no token bill | 70B+ prompt work — decode ~2–3 tok/s |
| Long-context / prompt ingest | Strong | Prefill is compute-bound; Spark digests fat prompts faster than it streams replies | Long completions on big models |
| Agent prototyping (tools, memory) | Excellent | Always-on local stack; NVIDIA agent tooling on DGX OS | Multi-region production |
| Code assist (7–20B) | Excellent | LMSYS: strong batching on 8B-class models | — |
| RAG pipeline dev (embed + retrieve + answer) | Strong | End-to-end on one box | Vector index at billions of chunks |
| Embeddings / reranking (moderate corpus) | Strong | Batch-friendly | Massive embedding farms |
| Offline batch (summarize, eval, synthetic) | Strong | N× single-node replicas | Petabyte-scale pipelines |
| Single-user 70B quantized chat | Moderate | Loads in 128 GB; decode ~2–3 tok/s (LMSYS 70B FP8) | Sub-2s first token at scale |
| Small-team interactive (~50 users) | Moderate | 2-node tensor parallel | 100+ concurrent SLO |
| High-volume API serving | Weak | Bandwidth + no MIG | H100/B200 fleets, many replicas |
| Large-scale training | Weak | Not the product | DGX Cloud / multi-GPU rack |
| Gaming / general CUDA | Poor | Wrong economics | Consumer GeForce |
Modality notes
RAG (embeddings, rerank, answer)
Build the pipeline on Spark: embed chunks with sentence-transformers or TEI, store vectors in a local index, rerank with a cross-encoder, answer with vLLM or SGLang. Embedding passes are batch-friendly; reranking is smaller batches. The limit is corpus size on disk and RAM for hot indexes, not whether GB10 can run the models.
Agents (tools, memory, always-on)
Agents loop: model proposes a tool call, runtime executes, results land back in context. Each round adds prefill on a growing prompt. Spark is a sane place to prototype tool schemas, memory policies, and failure modes before you pay datacenter rates — NVIDIA’s DGX Spark stack includes streamlined agent tooling (NemoClaw on recent DGX OS builds) for that local loop. Production multi-region routing is a different problem.
Speech (ASR / TTS)
Whisper-class ASR and lightweight TTS models fit comfortably in unified memory; realtime voice agents add decode latency on top of ASR + LLM + TTS chains. Spark can host the stack for dev; sub-100 ms voice loops at high concurrency need bandwidth Spark does not have.
Vision and multimodal
Classification and OCR models are often small relative to LLMs. VLMs (vision-language models) load image encoders beside text weights — memory-heavy, which is where 128 GB UMA helps. Video analytics sustained on many streams stress bandwidth and sustained decode similarly to LLM serving.
Diffusion (image / video)
Image diffusion (Stable Diffusion, Flux) loads large weight stacks and iterates denoise steps — memory for weights, bandwidth for the loop. Spark can run ComfyUI or similar for prototyping; farm-scale image generation at SLA is rack hardware.
When to cluster two Sparks
One Spark is enough for most dev: 8B–27B interactive, RAG prototyping, Whisper on the side. Add a second unit when a quantized 70B+ model needs tensor parallel across nodes, or when you want HA for a small interactive service — NVIDIA documents 405B-parameter support for a dual-Spark cluster over ConnectX-7 (200 Gb/s aggregate), again in FP4-class regimes aimed at prototype scale.
Clustering adds ops: QSFP links, NCCL over Ethernet, memory headroom on both nodes. Skip it until a single box fails on model size or per-user latency — not because marketing showed two pretty Sparks in a photo (although they are).
Frequently asked questions
Is chat the main inference workload on Spark?
Chat is the most visible sub-type, not the only one. Spark tenants also run code assistants, RAG stacks, embeddings, speech, and batch eval — often on the same box in sequence, not in parallel. Name the modality before you size the machine.
Can Spark replace production API serving?
For a high-volume public API with strict latency SLOs and hundreds of concurrent users, usually no. GB10 decode is bandwidth-limited and the GPU cannot be hardware-partitioned. Spark fits validation, staging, and small-team interactive paths; H100/B200-class fleets or DGX Cloud fit sustained production throughput.
What model sizes fit in 128 GB?
NVIDIA documents up to ~200B parameters on one Spark and ~405B on two, typically with FP4/NVFP4-class quantization. LMSYS ran Llama 3.1 70B (FP8) and GPT-OSS 120B (MXFP4) on a single unit — loads that choke most desktop GPUs. Fit is not speed: 70B decode on Spark was ~2.7 tok/s in their SGLang run.
Does fine-tuning count as inference?
No. Fine-tuning updates weights; inference reads fixed weights. Spark supports both on paper, but this post is about forward-pass workloads. Rack-scale training still belongs on multi-GPU servers or cloud.
Conclusion
Pick hardware after you name the workload: modality, concurrency, and whether the bottleneck is prefill or decode. Spark is a capacity-at-desktop GB10 box — 128 GB UMA, ~273 GB/s, no MIG — not a substitute for HBM bandwidth at datacenter scale.
Rent or buy Spark when you need local reproducibility, agent and RAG prototyping, code assist on 7–20B models, or offline batch on your own timeline. Move to DGX Cloud or Hopper/Blackwell racks when the job is rack-scale training or API serving at high concurrency.
If you want Spark without building the cluster and memory runbooks yourself, see spark.enverge.ai for DGX Spark pricing and the Enverge Cloud docs.
Glossary
| Term | Meaning |
|---|---|
| ASR | Automatic speech recognition |
| FP4 | 4-bit floating-point format; GB10 Tensor Cores accelerate FP4 inference |
| ITL | Inter-token latency — delay between generated tokens during decode |
| KV cache | Key-value cache; stored attention states during autoregressive decode |
| NVFP4 | NVIDIA’s FP4 quantization format (e.g. TensorRT Model Optimizer exports) |
| Prefill | Prompt-processing phase; parallel attention over all input tokens |
| RAG | Retrieval-augmented generation |
| SLO | Service level objective — e.g. a latency or throughput floor a deployment must meet |
| TP | Tensor parallelism — sharding one model across multiple GPUs or nodes |
| TTFT | Time to first token |
| TTS | Text-to-speech |
| UMA | Unified memory architecture; CPU and GPU share one physical memory pool |
| VLM | Vision-language model |