KV cache

Key-Value cache. The memory a transformer model uses to remember the conversation: every new token adds a small entry that grows with context length and the number of users served in parallel. It is the VRAM component that varies with your settings and must be accounted for per user.