Scaling self-hosted language models with distributed key-value caching

Self-hosting large language models often incurs high latency and redundant compute costs due to isolated GPU memory caches.

3 min read

Scaling self-hosted language models with distributed key-value caching

Running large language models on your own graphics processing units gives you complete control over costs, accuracy, and response times. However, self-hosting often requires paying for redundant work. When multiple servers handle chat applications or coding assistants, each server maintains its own isolated memory cache. These small caches cannot hold or share the growing context that modern artificial intelligence tools accumulate.

Consider a legal assistant that processes hundreds of thousands of tokens of case history in a single conversation. When a user returns to an existing thread, the new server handling the request lacks that history in its local memory. As a result, the graphics processor must recompute the key-value attention states for all past tokens before generating a single word. This prefill phase introduces significant delay, slowing down resumed chat sessions, document question-and-answering tools, and coding agents.

Key facts

Cost reduction
Up to 40%
Time to first token reduction
Up to 80%
Required LMCache version
0.5.8rc1 or later
Required vLLM version
0.11.1 or later
Architecture diagram showing distributed key-value caching setup with shared infrastructure connecting serving engines.
Diagram illustrating the distributed KV caching architecture and the proposed configuration for handling legal assistant workloads., Source ↗

Understanding distributed key-value caching

Distributed key-value caching extends local graphics memory with a shared tier placed alongside your hardware cluster. This allows any server to reuse key-value states across sessions, routing shifts, and server failures for days at a time. By connecting a shared cache tier using Amazon ElastiCache, teams can reduce self-hosted inference costs by up to 40 percent and cut the time to first token by up to 80 percent.

A new agent skill inside the Agent Toolkit for AWS helps developers evaluate and configure this setup through simple prompts. The skill works with coding assistants such as Kiro and Claude Code. It guides users through checking whether a specific workload benefits from shared caching, sizing a cache cluster for testing or production, and troubleshooting data transfers.

Evaluating potential savings

Calculating the benefits of a shared cache can be complex because savings depend on model types, user traffic, and prompt routing. To solve this, developers can use a companion tool called KVTrace to record memory cache events from serving engines like vLLM or SGLang. The recorder tracks block hashes and token counts without storing sensitive prompt text.

Table and text output detailing monthly cost comparisons and shard recommendations for L2 KV caching.
Cost comparison and calculations for adding an L2 KV cache to a server fleet based on specific workloads., Source ↗

The coding agent uses these trace logs to estimate cost savings before any infrastructure changes are made. It evaluates factors such as prompt length, checking whether the time saved by loading cached data exceeds the time required to read it from the network. If the projected savings do not cover the cost of the cache cluster, the agent halts the process.

Performance report table breaking down hit rates, saved tokens, and gross savings by capacity.
KVTrace report showing net savings across different cache hit shares and time-to-first-token benchmarks., Source ↗

Implementation and configuration

When setting up the infrastructure, the agent connects the LMCache connector to existing serving engines while preserving current routing rules. It ensures that all servers use matching key mechanisms and explicit data types to prevent key mismatches where one server writes data that another cannot locate.

Prerequisites for this setup include a compatible coding agent, graphics instances that support high-performance networking via Elastic Fabric Adapter remote direct memory access, LMCache version 0.5.8rc1 or later, vLLM version 0.11.1 or later, and ElastiCache for Valkey version 9.1 or later running on specific cluster nodes.

Source: Original announcement

Worth a read?

Report

Comments