Reducing token costs and latency in AI agent JSON workloads
Handling large structured inputs in multi-tool AI agents can cause token usage and response latency to spike.
3 min read, Wednesday, 18 November

Building production AI agents with dozens of tools and large structured inputs often introduces significant bottlenecks in token consumption and latency. When agents process large JavaScript Object Notation data, commonly known as JSON, every additional tool invocation injects the full payload into the large language model context window. This causes token usage and response latency to spike significantly. Over multi-turn conversations, this pressure compounds, driving up costs and degrading response quality. Effectively managing context window pressure in agentic systems is key to reliability and responsiveness at scale.
Key facts
- License
- MIT-licensed
- Hosting
- Cloud: Fully managed and hosted as part of LangSmith
- Hybrid: SaaS control plane, self-hosted data plane
- Fully Self-Hosted: Entirely on your own infrastructure
- Data Privacy
- Cloud: —
- Hybrid: No data leaves your VPC
- Fully Self-Hosted: —
- Provisioning and Scaling
- Cloud: Automatic updates and zero maintenance
- Hybrid: Managed as a service
- Fully Self-Hosted: —
| Metric | Cloud | Hybrid | Fully Self-Hosted |
|---|---|---|---|
| Hosting | Fully managed and hosted as part of LangSmith | SaaS control plane, self-hosted data plane | Entirely on your own infrastructure |
| Data Privacy | — | No data leaves your VPC | — |
| Provisioning and Scaling | Automatic updates and zero maintenance | Managed as a service | — |

Understanding the optimization challenge
Airline operations platforms often help manage crews, flights, and operations through specialized tools. A training schedule optimization agent built for this purpose verifies crew training schedule assignments against complex business rules. Equipped with over forty tools and inputs reaching megabytes in size, the agent answers complex queries across crew assignments, scheduling, and resource allocation. Although the agent delivered the required functionality, average request latencies exceeded fifteen seconds across most calls.
Every tool call injected its full payload into the model context, inflating token counts and costs. On larger datasets, the context window filled up before the agent could finish reasoning. Developers implemented three composable strategies to address these challenges. Together, these optimizations reduced token consumption by up to seventy-one percent and cut average latency by up to eighty percent across benchmark runs on a representative twenty-three megabyte payload.
Persisting lightweight session state
The first optimization changed how session state is persisted. Initially, large payloads were stored too close to the conversational state. Developers moved heavy artifacts, including request and response data along with validation reports, to object storage. Platform memory then persists only stable references to these objects, keeping the session state lightweight.

This approach relies on content addressing, which serializes the data with sorted keys and compact separators and uses the cryptographic secure hash of those bytes as the reference. Content addressing makes writes idempotent, meaning identical payloads map to the same key. A turn that re-serializes an unchanged request reuses the object already stored instead of uploading it again.
Paging large result lists on demand
Even after moving heavy payload persistence to object storage, tool responses remained a major token bottleneck. Many tool results were too large, which created two distinct problems. First, systems were sending full lists to the model even when it only needed a small subset. Second, those full list payloads were injected into context, rapidly consuming the available window. Implementing result paging allows agents to retrieve data incrementally on demand, preventing excessive data transfer during standard conversational turns.
Source: Original announcement
Worth a read?



Comments