Amazon SageMaker AI adds KV cache-aware routing for large language models
Amazon SageMaker AI has introduced a new routing strategy that tracks cache states directly from model containers.
2 min read, Wednesday, 18 November

Amazon SageMaker AI has introduced a new routing option called KV cache-aware routing, designed to help teams running large language models — artificial intelligence systems trained on vast amounts of text — handle multi-turn conversations and long prompts more efficiently. When serving large language models, applications often send repetitive text at the beginning of requests, such as long system prompts or chat histories. Models store the processed results of these prompts in GPU memory, known as the KV cache, so they can skip repeating that work on subsequent requests.
Key facts
- Tested model
- Llama 3.1 70B Instruct
- Instance type
- ml.p5.48xlarge
- Time to first token reduction
- up to 97 percent
- Throughput increase
- up to 234 percent
- Cache hit rate improvement
- 80 to 93 percent
- Source of cache location
- PREFIX_AWARE: Inferred from the request prefix
- KV_AWARE: Cache state reported by the container
- Matching mechanism
- PREFIX_AWARE: First PrefixLength bytes or characters
- KV_AWARE: Tokens, longest cached match
- Container requirements
- PREFIX_AWARE: Any container with prefix caching on
- KV_AWARE: A supported container with KV cache events on
| Metric | PREFIX_AWARE | KV_AWARE |
|---|---|---|
| Source of cache location | Inferred from the request prefix | Cache state reported by the container |
| Matching mechanism | First PrefixLength bytes or characters | Tokens, longest cached match |
| Container requirements | Any container with prefix caching on | A supported container with KV cache events on |

How the new routing strategy works
Unlike older methods that guess where a prompt might be cached based on fixed-length prefixes, this new approach relies on the model container itself. The container reports whenever it stores or evicts cache entries, allowing SageMaker AI to maintain a managed cache index of the current state across the endpoint. When a new request arrives, the endpoint tokenizes the prompt — breaking it down into smaller text pieces — and consults the index to find the instance holding the longest matching cache.

The system weighs how much of the prompt each instance has cached against how busy those instances currently are. If the best-matching instance is too busy, the request goes to the next available option, or falls back to standard load-aware routing if no useful match is found. This helps maintain high cache hit rates even when GPU memory fills up and instances begin evicting older cache entries.
Performance and comparisons
In benchmarks conducted with Llama 3.1 70B Instruct running on seven ml.p5.48xlarge instances over 30-minute tests, the new strategy reduced time to first token by up to 97 percent and increased overall throughput by up to 234 percent compared to random routing. It also raised the KV cache hit rate from three to 48 percent up to 80 to 93 percent. These gains are most pronounced during heavy workloads with long prompts, while shorter prompts see more modest improvements due to having less prefill data to save.

Source: Original announcement
Worth a read?



Comments