Welcome to Unified Cache Manager

Unified Cache Manager

Star Watch Fork

The core principle of Unified Cache Manager (UCM) is to persist the LLM KVCache and replace redundant computations through multiple retrieval mechanisms. UCM not only supports prefix caching but also offers a variety of training-free sparse attention retrieval methods, delivering higher performance when handling extremely long sequence inference tasks. Additionally, UCM provides a PD disaggregation solution based on a storage-compute separation architecture, which enables more straightforward and flexible management of heterogeneous computing resources. When integrated with vLLM, UCM achieves a 3-10x reduction in inference latency across various scenarios, including multi-turn dialogue and long-context reasoning tasks.

For more information, check out the following:

• UCM Section of the ModelEngine Community

Paper list:

• HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference

• ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding

• AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding

• Dynamic Early Exit in Reasoning Models

• Sparse Attention across Multiple-context KV Cache

Documentation

Getting Started

• Quickstart-vLLM
• Quickstart-vLLM-Ascend
• Quickstart-SGLang
• KV Cache Size Calculator

User Guide

• Feature and Model Support Matrix
• Prefix Cache
• Sparse Attention
• PD Disaggregation
• Observability
• Rectified Rotary Position Embeddings

Design Documents

• Store Architecture

Developer Guide

• UCM Contributing Guide
• Deep Dive into UCM
• How to Add A New Metric
• Extending UCM Store

About Us

• About Us