TL;DR
LMCache is an open-source framework that efficiently manages KV cache in LLM inference to reduce latency and increase throughput.
Key features
Persistently stores KV cache and enables reuse across multiple serving engines
Reduces TTFT (Time-to-First-Token) and improves throughput
Optimized for multi-turn conversations, RAG, and agent workloads
Vendor-neutral and integrable with vLLM, etc.
Supports distributed CPU memory sharing (P2P) and multimodal models
When to use it
When you need to reduce first response time in LLM services handling long contexts
For workloads that benefit from KV cache reuse, such as multi-turn conversations or RAG
When you want to share cache across multiple serving engines to reduce infrastructure costs