On-Prem LLM Infrastructure
What does it actually take to run a trillion-parameter model in your AI Lab for under $50k?
The Problem: Why On-Prem AI Now
For years, cloud-first was the default way to access large language models. That case has shifted: the economics, regulatory landscape, and capability of local GPU hardware have converged to make on-prem inference not just viable, but strategically superior for a growing set of enterprise workloads.
Four reasons organizations are pulling inference back on-prem:
Control
Own the runtime, the model weights, and the data path. No dependency on a third party's availability or policy changes.
Privacy
Sensitive IP, customer data, and regulated workloads never leave your perimeter. Compliance by architecture, not by contract.
Cost
High-volume inference at scale reaches cost parity with cloud in under 18 months on modern GPU hardware.
Flexibility
Run any open-weight model, fine-tune on proprietary data, and swap models without re-engineering the stack.
A framework for deciding what actually belongs on-prem:
Sensitivity
Which data classifications, workload types, or regulatory requirements make cloud inference a non-starter?
Volume
Where do high-frequency inference patterns make cloud egress and per-token pricing erode ROI at scale?
Customization
Which use cases require fine-tuning on proprietary data or model behavior that can't be achieved with a shared API?
Integration depth
Where does inference need to be deeply embedded in low-latency, synchronous operational pipelines that can't tolerate external API variability?
How It Works: The On-Prem Stack
Building a local inference platform means making deliberate choices at every layer:
GPU tiers and roles
Not every workload needs the same hardware, and matching tier to task is one of the highest-leverage decisions in an on-prem deployment:
Edge / dev tier
Small form-factor GPUs (RTX 4090, L4), ideal for developer sandboxes and small models under 13B parameters.
Inference tier
Mid-range multi-GPU nodes (DGX Spark 128GB, RTX Pro 6000, Mac Studio 256GB), suited to 13B–70B models at moderate scale.
Training / large-inference tier
High-memory nodes (B200, B300) for 800B+ models, fine-tuning runs, and multi-modal workloads.
Model Serving Patterns & Selection
Start with the simplest pattern that meets your SLA; complexity is easy to add later and hard to remove:
Static endpoint
A single model, a fixed endpoint, low overhead.
Router / gateway
A routing layer dispatches requests to the right model by task type or cost, enabling a multi-model fleet behind one API.
Agent runtime
Inference embedded inside an orchestration loop, with models calling tools and iterating on feedback.
Model selection criteria — fit for task beats model size ego; a well-prompted 13B model will often outperform a poorly scaffolded 70B model in production, at a fraction of the cost. Weigh: task fit, context window needs, latency budget, and license/governance requirements (open-weight does not mean open-use — validate commercial licensing and data residency terms before production).
Quantization and NVFP4 — quantization reduces model weight precision to shrink memory footprint and increase throughput with minimal quality loss. NVIDIA's NVFP4 format (introduced with Blackwell) can cut memory footprint up to 4x versus FP16, roughly double throughput on supported hardware, and typically costs less than 2% quality degradation on standard evals when well-calibrated — meaning a cluster that used to top out at a 70B model can serve something in the hundreds-of-billions range without a hardware refresh.
Tech Stack
Cluster Hardware
8x NVIDIA DGX Spark cluster — GB10 (sm_121, consumer-Blackwell family), aarch64, 128 GB unified memory per node
Networking
200G RoCE fabric via ConnectX-7, dual-rail cabling for bandwidth headroom
Serving & Orchestration
vLLM — both NVIDIA's Spark-specific build and upstream, depending on the model. Ray for cluster orchestration, Ansible for fabric/NCCL validation and launch automation
Storage
Ceph for model/data storage, staged to local NVMe cache per node
Models Tested
nvidia/GLM-5.2-NVFP4 (744B total / 40B active parameters, DeepSeek-style sparse attention) and nvidia/Kimi-K2.7-Code-NVFP4 (1T total / 32B active parameters, code-focused, dense attention)
Quantization
NVFP4 quantization on both checkpoints, with attention/embeddings kept in higher precision
Findings: What Mark Built and Learned
(Honest assessment — matches the candid tone of the other prototype pages. This is dense, technical material; the headline lessons are worth reading even if you skip the details.)
Getting a 744-billion-parameter model to serve at all took eight attempts. Mark got GLM-5.2-NVFP4 — a 744B-parameter mixture-of-experts model with a sparse attention mechanism — running across all eight DGX Sparks at full size, apparently one of the first times this model had run on this class of hardware without pruning it down first. None of the mainstream sparse-attention kernels were built for the Spark's chip (a consumer-Blackwell part, not the datacenter-class chips they're normally tuned for), so getting there meant patching three separate kernel families, porting open-source community fixes with clean fallbacks so nothing breaks on other hardware. The failures were deceptive the whole way: the cluster would form, weights would load, health checks would pass green — and then the engine would die the moment it got a real request, because the startup checks never actually exercised the part that was broken.
The scariest bug wasn't a crash — it was silence.
After the initial bring-up, performance tuning uncovered something worse than an outage: the cluster had been silently generating garbage on every request longer than about 2,000 tokens, while passing every health check and benchmark the whole time. The root cause was a subtle mismatch between the model's architecture (which expects most of its attention layers to simply reuse a neighboring layer's calculations) and the serving software (which didn't know about that shortcut, and instead ran those layers on uninitialized data). Every parallel worker in the cluster ended up selecting a different, effectively random set of tokens to pay attention to — and averaged its answer with the others. Short prompts and quick test chats never hit enough tokens to expose it. The fix was small (about 30 lines of code) but finding it required building custom instrumentation into a live model, since the bug was invisible to every standard test.
Once fixed, targeted tuning nearly doubled decode speed (roughly 9 to 15+ tokens/second), and the single biggest lever wasn't networking or GPU graph tuning — it was speculative decoding (a technique where the model drafts several tokens at once and verifies them together), worth a 53% speed gain on its own.
A second, even larger model told a different story.
Later, the team hit instability running GLM-5.2 at very long context under real workloads, and rather than just scaling back, used it as a prompt to try a fundamentally different model: Kimi-K2.7-Code, a full 1-trillion-parameter model (with only 32B "active" at a time — a different efficiency trade than GLM-5.2). It has no sparse-attention shortcut (so the GLM-style correctness trap doesn't exist for it) but also no speculative-decoding support (so that performance lever isn't available either). Getting it running hit four separate bring-up failures before serving its first token, including a memorable one where two of the eight cluster nodes were quietly loading data 10–20x slower than their peers — initially mistaken for a software bug, until simply rebooting those two machines fixed it completely.
The model's advertised limits didn't hold up under testing. Kimi-K2.7-Code claims support for roughly 260,000 tokens of context. Systematic testing found a hard ceiling well short of that: reliable, correct answers up to about 127,000 tokens, and failures beginning around 252,000 tokens — even in the "easy" case of information placed near the end of the prompt, where a model should have the best chance of getting it right. Rather than chase the advertised number, the deployment was deliberately capped at 128,000 tokens: a proven-good length with headroom below the failure zone, prioritizing reliability over a bigger number on paper.
Matching software to hardware mattered more than using the newest version. An initial version of the serving stack worked correctly but crashed unpredictably roughly once a day under ordinary use. Switching to the hardware vendor's own tested software image — slightly older, but built and validated specifically for this chip — resolved the instability.
The bottom line was counterintuitive: the newer, much larger 1-trillion-parameter model ended up running faster in production (about 19 tokens/second) than the earlier 744-billion-parameter model (about 15 tokens/second, after tuning) — without using any of the speed-up tricks that made the smaller model fast. Parameter count alone doesn't predict real-world throughput; architecture does.
What This Means for Your Organization
Running frontier-scale, even trillion-parameter-class, open-weight models on-prem is achievable on a modest cluster — not a warehouse-scale datacenter — but "it's running" and "it's correct" are two different bars, and only one of them shows up in a health check.
Long-context correctness testing (deliberately hiding a known answer deep in a long prompt and confirming the model finds it) needs to be part of any go-live checklist for a serious LLM deployment — short test chats will pass on a broken system.
Model selection should weigh architecture-specific tradeoffs — sparse vs. dense attention, speculative-decoding support, advertised vs. actually-proven context length — as heavily as raw benchmark scores or parameter counts.
Matching your serving software to your hardware vendor's own validated stack, rather than always chasing the newest release, can be the difference between a stable deployment and a flaky one.
Resources
Full technical write-ups are available on the nvisia AI Lab blog:
GLM-5.2 (744B) Serving on 8× DGX Spark — the Full Bring-Up Story
GLM-5.2 on 8× DGX Spark, Part 2 — Performance Headroom, and the Bug That Ate Our Benchmarks
Kimi-K2.7-Code on 8× DGX Spark — When the Fix Is a Different Model