Overcoming Confidential Computing Overheads: Optimizing Private LLM Inference on NVIDIA Blackwell
突破機密運算效能瓶頸：NVIDIA Blackwell 與 TensorRT-LLM 的隱私推理優化
As LLMs process sensitive data, Confidential Computing (CC) with memory-encrypted VMs and confidential GPUs is essential. However, secure hardware execution alters data movement and synchronization, risking performance loss. NVIDIA's engineering team benchmarked DeepSeek-R1 on eight B200 GPUs to show that with CC-aware adaptations in TensorRT-LLM—such as asynchronous readbacks and GPU %globaltimer measurements—private inference can retain over 96% of non-CC throughput with under 5% latency overhead, proving production-ready security without sacrificing speed.
Key points
Near-Baseline Throughput
Running DeepSeek-R1 on eight B200 GPUs retains 96.1% to 98.2% of the unencrypted baseline throughput when Confidential Computing is enabled.
Minimal Latency Penalty
The mean Time Per Output Token (TPOT) latency overhead is kept within a minimal 1.2% to 4.3% range.
Optimized Data Paths
Overcomes encryption bounce buffer bottlenecks by selecting pageable memory and using asynchronous workers for token and sampling-data readback.
Stable Kernel Autotuning
Solves unstable CUDA-event timing under CC by switching to the GPU’s %globaltimer for accurate kernel tactic selection.
Adaptive Multi-GPU Comm
Detects the absence of NVLS multicast under CC and dynamically selects optimal NCCL communication algorithms to minimize latency.
How it works
| 標準環境 (CC Off) | 機密運算環境 (CC On) | TensorRT-LLM 優化策略 | |
|---|---|---|---|
| Host-Device Transfer | 透過固定記憶體進行非同步快速傳輸 (Fast async via pinned memory) | 需通過加密暫存區（Bounce Buffer）並阻塞執行緒 (Blocked by software-encrypted bounce buffer) | 改用分頁記憶體，將 Token 讀回移至非同步背景執行緒 (Async worker readback & pageable memory) |
| Kernel Autotuning | 使用穩定的 CUDA 事件時間戳記 (Stable CUDA event timestamps) | CUDA 事件計時訊號不穩定，可能選錯算子 (Unstable CUDA event timing, risks poor tactic selection) | 強制改用 GPU 全域計時器 %globaltimer 進行測量 (Switched to GPU %globaltimer for precision) |
| Multi-GPU Comm | 支援高速 NVLS 多播技術 (Fast NVLS multicast enabled) | NVLS 不可用，NCCL 同步開銷增加 (NVLS unavailable, increased NCCL sync overhead) | 自動偵測 NVLS 狀態並適應性選擇通訊演算法 (Detect NVLS absence & optimize collective path) |
Why it matters
Traditionally, robust security came at a steep performance cost. As enterprises deploy generative AI with sensitive proprietary data, this co-optimization of Blackwell hardware and TensorRT-LLM proves that confidential computing is ready for production. It removes the security barrier for highly regulated sectors like finance, healthcare, and government, enabling them to run state-of-the-art models securely without compromising on extreme speed.
Who it affects
- AI Developer
- Enterprise Leader
- AI Researcher
- Product Manager
How to use it
- 1Financial and healthcare organizations running private data inference and compliance audits inside secure CVMs.
- 2Hardware-level protection of sensitive enterprise intellectual property and proprietary model weights across multi-GPU clusters.
Limitations & caveats
- NVLS (NVLink SHARP) multicast is unavailable in B200 CC configurations, limiting certain multi-GPU communication optimizations.
- Legacy inference frameworks without specific CC-aware adaptations will experience substantial performance degradation due to secure execution paths.
Related
NVIDIA Launches Open-Source NVCRE: Automating GPU Cluster Readiness Validation Before AI Workloads Run
NVIDIA 開源 NVCRE：在 AI 工作負載上線前，自動驗證 GPU 叢集準備狀態
Traditional GPU health checks often miss performance bottlenecks in distributed training. NVIDIA's open-source NVCRE is a Kubernetes controller that runs active, topology-aware workloads to pinpoint failing nodes before production starts.