AI21 Nov 20262 min read

AWS toolkit cuts self-hosted LLM latency up to 80% with distributed KV caching

AWS released a coding agent skill for distributed KV caching that can reduce self-hosted AI inference costs by up to 40% and cut time to first token by up to 80%.

Listen to this storyAbout 3 min, read by your device
Picture made with AI

Self-hosting large language models gives developers control over infrastructure, but long conversations often trigger slow and expensive recalculations. When an AI agentAI agentAI software that plans a task, uses tools and checks its own result. Open in glossary or multi-turn chat application resumes after an idle period, local GPUGPUA chip that does many small sums at once. First made for graphics, now the main chip for AI. Open in glossary memory often lacks the prior conversation state. The server must then recompute key-value attention states for the entire context before generating text. This cold prefill stage can leave users waiting minutes on prompts with tens of thousands of tokens.

Shared storage across GPU clusters

AWS has introduced a distributed KV caching skill in its Agent Toolkit to tackle this bottleneck. Distributed caching pools key-value attention states into a shared tier, allowing any GPU server in a cluster to reuse previously calculated states across different sessions and worker failures.

According to AWS, combining this setup with Amazon ElastiCache can lower self-hosted inferenceInferenceRunning a trained AI model to get an answer. Open in glossary expenses by as much as 40% and slash time to first tokenTime to first tokenThe duration between sending a prompt to an AI model and receiving its very first generated word. Open in glossaryTokenA small piece of text, about three quarters of a word, that an AI model reads or writes. Open in glossary by up to 80%. The new tool allows developers to evaluate, size, and deploy this infrastructure through coding assistants like Claude Code, Kiro, and OpenAI Codex.

Automated evaluation and deployment

The skill guides an agent to check if a workload will actually benefit before any infrastructure is provisioned. For short prompts, loading attention state from a network cache can take longer than recalculating it locally. The agent identifies this cutover point and checks whether expected savings justify the hardware costs.

  • Inspects GPU fleets using read-only calls to verify network interfaces like Elastic Fabric Adapter.
  • Uses an offline tool called KVTrace to replay promptPromptThe text you give an AI model to tell it what to do. Open in glossary traffic and measure potential cache hit rates without storing raw text or token IDs.
  • Connects LMCache to existing serving engines, including vLLM and SGLang.
  • Verifies that identical prompts generate the same outputs from the cache to avoid configuration mismatches.

The tool requires GPU instances that support RDMA writes, such as AWS p5 instances, and links with ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes.

How does this story land?

One tap per reader.

Quick poll

Are you using distributed caching to speed up your self-hosted AI workloads?

One tap. No sign-up.

Ask this story

Still have a question?

Answers use only what is in this story. They are written by AI and can be wrong.

Sources

  1. Artificial Intelligence, Cut self-hosted LLM cost and latency with the distributed KV caching agent skill
Companies in this storyAmazonWords explainedAI agentGPUTime to first tokenInferenceTokenPrompt
Up next in AIAWS Quick lets workers build multi-agent workflows directly from chat2 min read

Keep reading

All AI stories