Amazon SageMaker AI adds KV cache-aware routing for large language models

Amazon SageMaker AI has introduced a new routing strategy that tracks cache states directly from model containers.

2 min read

Amazon SageMaker AI adds KV cache-aware routing for large language models

Amazon SageMaker AI has introduced a new routing option called KV cache-aware routing, designed to help teams running large language models — artificial intelligence systems trained on vast amounts of text — handle multi-turn conversations and long prompts more efficiently. When serving large language models, applications often send repetitive text at the beginning of requests, such as long system prompts or chat histories. Models store the processed results of these prompts in GPU memory, known as the KV cache, so they can skip repeating that work on subsequent requests.

Key facts

Tested model
Llama 3.1 70B Instruct
Instance type
ml.p5.48xlarge
Time to first token reduction
up to 97 percent
Throughput increase
up to 234 percent
Cache hit rate improvement
80 to 93 percent
Routing strategies on Amazon SageMaker AI
Source of cache location
PREFIX_AWARE: Inferred from the request prefix
KV_AWARE: Cache state reported by the container
Matching mechanism
PREFIX_AWARE: First PrefixLength bytes or characters
KV_AWARE: Tokens, longest cached match
Container requirements
PREFIX_AWARE: Any container with prefix caching on
KV_AWARE: A supported container with KV cache events on
JSON response snippets illustrating the increase in cached tokens between the first request and a follow-up request.
Comparison of API responses showing an initial request with minimal cached tokens versus a follow-up request with high prompt token reuse from the KV cache., Source ↗

How the new routing strategy works

Unlike older methods that guess where a prompt might be cached based on fixed-length prefixes, this new approach relies on the model container itself. The container reports whenever it stores or evicts cache entries, allowing SageMaker AI to maintain a managed cache index of the current state across the endpoint. When a new request arrives, the endpoint tokenizes the prompt — breaking it down into smaller text pieces — and consults the index to find the instance holding the longest matching cache.

Diagram showing the KV cache-aware routing mechanism between client applications, endpoint routers, managed cache indices, and model instances.
Architecture diagram showing how the endpoint router consults the managed cache index to direct requests to the model container instance with the longest matching cache., Source ↗

The system weighs how much of the prompt each instance has cached against how busy those instances currently are. If the best-matching instance is too busy, the request goes to the next available option, or falls back to standard load-aware routing if no useful match is found. This helps maintain high cache hit rates even when GPU memory fills up and instances begin evicting older cache entries.

Performance and comparisons

In benchmarks conducted with Llama 3.1 70B Instruct running on seven ml.p5.48xlarge instances over 30-minute tests, the new strategy reduced time to first token by up to 97 percent and increased overall throughput by up to 234 percent compared to random routing. It also raised the KV cache hit rate from three to 48 percent up to 80 to 93 percent. These gains are most pronounced during heavy workloads with long prompts, while shorter prompts see more modest improvements due to having less prefill data to save.

Line graph showing lower latency for KV_AWARE routing compared to RANDOM routing over time.
CloudWatch chart displaying one-minute average model latency for random routing versus KV-aware routing., Source ↗

Source: Original announcement

Worth a read?

Report

Comments