1.6.1
The Standard Attention Memory Problem
What is it?
Standard attention computes an attention score matrix from query and key representations, applies softmax, and multiplies the result by value representations. For a sequence of length n, the attention matrix contains n × n values. Storing and repeatedly accessing this matrix in high-bandwidth memory (HBM) creates a major memory bottleneck for long sequences.
Key points
1. Standard attention calculates attention scores using QK transpose.
2. The score matrix has dimensions n × n for a sequence of length n.
3. Softmax is applied to the attention scores.
4. The resulting weights are multiplied by V.
5. The full attention matrix can require O(n²) memory.
6. Writing and reading intermediate results from HBM creates significant memory traffic.
7. The problem becomes more severe as sequence length increases.
Example: Attention memory at long sequence lengths
Consider a sequence containing 8,192 tokens.
The attention matrix contains:
8,192 × 8,192 = 67,108,864 values.
With 16-bit values, this is approximately 128 MiB per attention head.
With 32 heads, the corresponding total is approximately 4 GiB per layer if all those matrices are materialized.
This illustrates why storing full attention matrices becomes expensive for long sequences.
Real-world application
Understanding the memory bottleneck helps engineers choose efficient attention implementations when training or serving long-context language models.
Key takeaway
The main limitation of standard attention for long sequences is the memory required to materialize and access the full attention matrix.