San Jose – November 06, 2026 -- Supermicro, IBM and NVIDIA have launched a joint KV cache storage solution that delivered up to 56x faster Time-to-First-Token (TTFT) at 130,000-token prompts in testing with a 120-billion-parameter large language model.
Test results show TTFT rising 0.377 seconds with cache versus 29.55 seconds without
As input length grew from 30,000 to 130,000 tokens, TTFT increased by only 0.377 seconds with persistent KV cache. Without it, TTFT increased by 29.55 seconds. The figures come from joint testing by Supermicro and NVIDIA and are detailed in an IBM Redbook.
The companies also ran cluster-level validation covering high-concurrency and noisy-neighbor network traffic scenarios.
Four vendors' components form the persistent storage tier
The architecture combines Supermicro Petascale all-flash storage servers built on AMD EPYC 9005 Series ("Turin") processors, IBM Storage Scale software, NVIDIA Dynamo inference software and NVIDIA Spectrum-X Ethernet networking. It adds a shared, high-capacity layer that holds KV cache content outside GPU and CPU memory.
Inference runs in two phases, prefill and decode. During prefill, input text is converted into tokens and processed to generate the KV states needed for inference. A persistent cache lets the system reuse those states rather than recompute the same context for later requests.
Agentic and long-context workloads drive demand for cache offload
Agentic workflows, coding assistants, long-context RAG, multimodal AI and multi-turn applications generate large KV state as they process codebases, documents, conversations and images. That state quickly exceeds GPU and CPU memory, creating the need for shared network storage.
According to the companies, extending cache capacity beyond local GPU memory reduces GPU idle time, raises inference throughput and supports greater concurrency without proportionally adding GPU resources. They also cite high availability of cached content, shared access across GPU nodes and clusters, and non-disruptive capacity scaling.
Supermicro CEO Charles Liang said enterprises and neoclouds running high-throughput inference "need to remove bottlenecks to improve GPU efficiency."
Supermicro rack integration cuts deployment from weeks to days
Supermicro's factory rack integration and on-site deployment services can shorten rollout of the solution from weeks to days, according to the company.
IBM Storage General Manager Sam Werner said the offering lets customers maximize GPU investments and scale storage independently of compute. NVIDIA AI Inference Product Lead Shruti Koparkar said Dynamo orchestrates distributed inference so enterprises extract more usable throughput from every GPU.