Decode
Model typesThe second phase of inference: the model generates the response token by token. Each token requires re-reading all model weights and the KV cache from memory, so the speed (tokens per second) is driven mostly by memory bandwidth.
The second phase of inference: the model generates the response token by token. Each token requires re-reading all model weights and the KV cache from memory, so the speed (tokens per second) is driven mostly by memory bandwidth.