Scaling self-hosted language models with distributed key-value caching
Self-hosting large language models often incurs high latency and redundant compute costs due to isolated GPU memory caches.
3 min read, Saturday, 21 November

Running large language models on your own graphics processing units gives you complete control over costs, accuracy, and response times. However, self-hosting often requires paying for redundant work. When multiple servers handle chat applications or coding assistants, each server maintains its own isolated memory cache. These small caches cannot hold or share the growing context that modern artificial intelligence tools accumulate.
Consider a legal assistant that processes hundreds of thousands of tokens of case history in a single conversation. When a user returns to an existing thread, the new server handling the request lacks that history in its local memory. As a result, the graphics processor must recompute the key-value attention states for all past tokens before generating a single word. This prefill phase introduces significant delay, slowing down resumed chat sessions, document question-and-answering tools, and coding agents.
Key facts
- Cost reduction
- Up to 40%
- Time to first token reduction
- Up to 80%
- Required LMCache version
- 0.5.8rc1 or later
- Required vLLM version
- 0.11.1 or later

Understanding distributed key-value caching
Distributed key-value caching extends local graphics memory with a shared tier placed alongside your hardware cluster. This allows any server to reuse key-value states across sessions, routing shifts, and server failures for days at a time. By connecting a shared cache tier using Amazon ElastiCache, teams can reduce self-hosted inference costs by up to 40 percent and cut the time to first token by up to 80 percent.
A new agent skill inside the Agent Toolkit for AWS helps developers evaluate and configure this setup through simple prompts. The skill works with coding assistants such as Kiro and Claude Code. It guides users through checking whether a specific workload benefits from shared caching, sizing a cache cluster for testing or production, and troubleshooting data transfers.
Evaluating potential savings
Calculating the benefits of a shared cache can be complex because savings depend on model types, user traffic, and prompt routing. To solve this, developers can use a companion tool called KVTrace to record memory cache events from serving engines like vLLM or SGLang. The recorder tracks block hashes and token counts without storing sensitive prompt text.

The coding agent uses these trace logs to estimate cost savings before any infrastructure changes are made. It evaluates factors such as prompt length, checking whether the time saved by loading cached data exceeds the time required to read it from the network. If the projected savings do not cover the cost of the cache cluster, the agent halts the process.

Implementation and configuration
When setting up the infrastructure, the agent connects the LMCache connector to existing serving engines while preserving current routing rules. It ensures that all servers use matching key mechanisms and explicit data types to prevent key mismatches where one server writes data that another cannot locate.
Prerequisites for this setup include a compatible coding agent, graphics instances that support high-performance networking via Elastic Fabric Adapter remote direct memory access, LMCache version 0.5.8rc1 or later, vLLM version 0.11.1 or later, and ElastiCache for Valkey version 9.1 or later running on specific cluster nodes.
Source: Original announcement
Worth a read?



Comments