Topology-Aware Workload Scheduling with NVIDIA Topograph
NVIDIA Topograph：實現拓撲感知排程，徹底釋放 AI 工廠 GPU 效能
Distributed AI workloads rely heavily on high-bandwidth communication. NVIDIA Topograph automates cluster topology discovery, eliminating the need for manual snapshots. It gathers physical interconnect data (such as NVLink and InfiniBand) from cloud APIs or on-premises networks and dynamically generates Kubernetes node labels or Slurm configurations. Working with KAI Scheduler or Slinky, it enables schedulers to place tightly coupled jobs within the closest physical GPU domains, reducing network contention and boosting efficiency.
Key points
Automated Topology Discovery
Automatically queries physical network and hardware connections from cloud APIs or on-prem networks, removing the need for manual mapping.
Multi-Scheduler Translation
Normalizes discovered data and outputs it directly as Kubernetes node labels, Slurm topology.conf, or Slinky ConfigMaps.
Maximizing AI Factory Efficiency
Enables schedulers to keep tightly coupled jobs within high-bandwidth NVLink domains, preventing idle GPUs from wasting provisioned power.
Event-Driven Dynamic Updates
Monitors cluster changes via Node Observer to refresh topology views in real time, featuring built-in aggregation delay to prevent request storms.
How it works
Why it matters
As AI models scale exponentially, power-limited AI factories face severe bandwidth bottlenecks. Without topology awareness, schedulers might scatter tightly coupled tasks across different racks, forcing traffic through shared links, causing network congestion, and wasting expensive GPU cycles. Topograph provides real-time, accurate physical network maps, allowing orchestrators like KAI Scheduler to perform topology-aware gang scheduling, significantly maximizing tokens-per-watt and reducing overall operation costs.
Who it affects
- AI Developer
- Enterprise Leader
- Product Manager
How to use it
- 1Enable topology-aware gang scheduling for LLM training and inference using KAI Scheduler or Kueue in Kubernetes.
- 2Automatically update physical topology configs for Slurm to optimize job allocations on bare-metal GPU supercomputers.
- 3Synchronize network topologies to ConfigMaps when running Slurm on Kubernetes via Slinky.
Limitations & caveats
- Highly dependent on cloud provider APIs or specific fabric tools (like ibnetdiscover or NetQ) to report topology information.
- Reflects reported physical topology rather than intended design, and refresh responsiveness is constrained by provider triggering events.
Related
Hugging Face Launches @huggingface/kernels: Over 200 Optimized WebGPU Kernels for Local Web AI
Hugging Face 推出 @huggingface/kernels：為網頁端在地 AI 提供超過 200 個極速 WebGPU 核心
Hugging Face released @huggingface/kernels and Fleet, a browser benchmarking suite, offering 207 optimized WebGPU kernels that outperform ONNX Runtime Web by 2.57x on Apple M4 GPUs.