Running Nvidia Dynamo on Amazon SageMaker HyperPod Inference

Amazon SageMaker HyperPod Inference now integrates NVIDIA Dynamo to support distributed inference for large language models.

2 min read

Running Nvidia Dynamo on Amazon SageMaker HyperPod Inference

Serving large language models efficiently requires careful management of GPU memory, prompt processing, and token generation. Large model weights often exceed the memory capacity of a single machine before accounting for runtime overhead and cache storage. To address these hurdles, Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining distributed inference capabilities with managed infrastructure.

Key facts

Version
1.0
Line chart comparing output tokens per second per GPU for aggregated versus disaggregated serving across user token targets.
Throughput comparison chart showing disaggregated serving achieving 207 tokens per second per GPU compared to 101 for aggregated serving on Qwen3-235B., Source ↗

Understanding the Integration

NVIDIA Dynamo is an open-source framework designed for distributed model serving. It supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing across engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments using high-performance networking through the Elastic Fabric Adapter, along with coordinated GPU scheduling and automated node recovery.

The HyperPod Inference add-on installs custom resource definitions and orchestration tools. These include an inference operator for endpoint configuration, a Dynamo operator for model workers and replicas, and a scheduling component that uses gang scheduling to place multi-node workers together reliably.

Deployment Architectures

Architecture diagram of NVIDIA Dynamo on SageMaker HyperPod showing application load balancing, operators, prefill and decode worker pools, and model weights.
Architecture diagram illustrating NVIDIA Dynamo running on a SageMaker HyperPod cluster with disaggregated prefill and decode worker pools., Source ↗

Teams can choose between aggregated serving, where prompt processing and token generation run on the same workers, or disaggregated serving, which separates them into independent pools. Disaggregation allows teams to scale prompt processing and token generation separately, passing the key-value cache between workers over high-speed networking.

The integration offers two primary configuration paths. Users can bring their own custom deployment manifests to retain direct control over the serving graph, or they can use managed endpoint configurations with standard OpenAI-compatible interfaces. Both paths support single-instance and multi-node setups while maintaining automated node health monitoring.

Performance Benchmarks

Benchmarks show distinct performance advantages for distributed setups. For instance, testing with a large model demonstrated that disaggregated serving achieved double the throughput of aggregated serving while meeting strict latency targets for time to first token and generation speed. Additional testing on multi-node configurations revealed that splitting a model across multiple instances retains over ninety percent of single-node throughput, proving that network boundaries introduce minimal performance penalties when properly configured.

Bar chart showing single-node versus two-node aggregated serving throughput metrics.
Bar chart comparing single-node and two-node aggregated serving throughput for Qwen3.5-397B, retaining 99.1 percent of performance., Source ↗

Source: Original announcement

Worth a read?

Report

Comments