Running Nvidia Dynamo on Amazon SageMaker HyperPod Inference
Amazon SageMaker HyperPod Inference now integrates NVIDIA Dynamo to support distributed inference for large language models.
2 min read, Wednesday, 18 November

Serving large language models efficiently requires careful management of GPU memory, prompt processing, and token generation. Large model weights often exceed the memory capacity of a single machine before accounting for runtime overhead and cache storage. To address these hurdles, Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining distributed inference capabilities with managed infrastructure.
Key facts
- Version
- 1.0

Understanding the Integration
NVIDIA Dynamo is an open-source framework designed for distributed model serving. It supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing across engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments using high-performance networking through the Elastic Fabric Adapter, along with coordinated GPU scheduling and automated node recovery.
The HyperPod Inference add-on installs custom resource definitions and orchestration tools. These include an inference operator for endpoint configuration, a Dynamo operator for model workers and replicas, and a scheduling component that uses gang scheduling to place multi-node workers together reliably.
Deployment Architectures

Teams can choose between aggregated serving, where prompt processing and token generation run on the same workers, or disaggregated serving, which separates them into independent pools. Disaggregation allows teams to scale prompt processing and token generation separately, passing the key-value cache between workers over high-speed networking.
The integration offers two primary configuration paths. Users can bring their own custom deployment manifests to retain direct control over the serving graph, or they can use managed endpoint configurations with standard OpenAI-compatible interfaces. Both paths support single-instance and multi-node setups while maintaining automated node health monitoring.
Performance Benchmarks
Benchmarks show distinct performance advantages for distributed setups. For instance, testing with a large model demonstrated that disaggregated serving achieved double the throughput of aggregated serving while meeting strict latency targets for time to first token and generation speed. Additional testing on multi-node configurations revealed that splitting a model across multiple instances retains over ninety percent of single-node throughput, proving that network boundaries introduce minimal performance penalties when properly configured.

Source: Original announcement
Worth a read?



Comments