KV Cache Calculator

⚙️ Configuration

Model Source

Preset Models

Custom Model

Select Model

Data Type

Number of Tokens

Batch Size

Tensor Parallelism (TP)

Data Parallelism (DP)

Maximum Tokens Calculator

Single-GPU Memory for KV Cache (GB)

Memory available for KV cache (excluding model weights)

📊 Results

📊

Configure your model and click calculate to see results.

📚 Formula Reference

🔸 MLA (Multi-head Latent Attention)

Single-GPU KV Cache = layers × tokens × batch × (kv_lora_rank + qk_rope_head_dim) / tp × dtype_size

Note: No factor of 2 because K and V are compressed together into latent space

kv_lora_rank: Compressed KV dimension

qk_rope_head_dim: RoPE positional encoding dimension

Models: DeepSeek V3/R1/V3.2, Kimi K2/K2.5, GLM-5

🔹 GQA (Grouped-Query Attention)

Single-GPU KV Cache = 2 × layers × tokens × batch × kv_heads × head_dim / tp × dtype_size

2: Key and Value matrices

kv_heads: Number of KV heads (less than attention heads)

head_dim: Dimension per attention head

Models: Qwen3, GLM-4.7, GLM-4.5, MiniMax-M2 series

⚠️ Hybrid (Mixed Architecture)

Single-GPU KV Cache = 2 × attn_layers × tokens × batch × kv_heads × head_dim / tp × dtype_size

Warning: Calculation may not be accurate for hybrid architectures. Further adaptation needed.

Linear + Full Attention: layer_types contains "linear_attention" and "full_attention" (e.g., Qwen3.5 series)

Sliding + Full Attention: layer_types contains "sliding_attention" and "full_attention" + sliding_window (e.g., Gemma4, GPT-OSS)

Attention + SSM: hybrid_layer_pattern with 0=SSM, 1=Attention (e.g., MiMo-V2-Flash)

Models: Qwen3.5 series

🔹 Standard Transformer (MHA/MQA)

Single-GPU KV Cache = 2 × layers × tokens × batch × hidden_size × (kv_heads / attn_heads) / tp × dtype_size

MHA: kv_heads = attn_heads (every head has its own KV)

MQA: kv_heads = 1 (all heads share single KV)

Models: GPT-2 (MHA), PaLM/BLOOM (MQA)

🔍 How to Identify Model Type from Config

1. MLA: Check for kv_lora_rank + qk_rope_head_dim

2. Hybrid: layer_types with multiple types, or hybrid_layer_pattern, or sliding_window with mixed layers

3. GQA: Has head_dim and kv_heads < attn_heads

4. Standard: Falls through to MHA/MQA/GQA based on kv_heads ratio

Multimodal: Config nested in text_config

⚙️ Configuration

Maximum Tokens Calculator

📊 Results

📝 Calculation Details

📚 Formula Reference