OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Flash Attention: Algorithm and Hardware Awareness
Lesson 39 of 415 6 min read
1.6.1

The Standard Attention Memory Problem

What is it?

Standard attention computes an attention score matrix from query and key representations, applies softmax, and multiplies the result by value representations. For a sequence of length n, the attention matrix contains n × n values. Storing and repeatedly accessing this matrix in high-bandwidth memory (HBM) creates a major memory bottleneck for long sequences.

Key points

1. Standard attention calculates attention scores using QK transpose. 2. The score matrix has dimensions n × n for a sequence of length n. 3. Softmax is applied to the attention scores. 4. The resulting weights are multiplied by V. 5. The full attention matrix can require O(n²) memory. 6. Writing and reading intermediate results from HBM creates significant memory traffic. 7. The problem becomes more severe as sequence length increases.

Example: Attention memory at long sequence lengths

Consider a sequence containing 8,192 tokens. The attention matrix contains: 8,192 × 8,192 = 67,108,864 values. With 16-bit values, this is approximately 128 MiB per attention head. With 32 heads, the corresponding total is approximately 4 GiB per layer if all those matrices are materialized. This illustrates why storing full attention matrices becomes expensive for long sequences.

Real-world application

Understanding the memory bottleneck helps engineers choose efficient attention implementations when training or serving long-context language models.
Key takeaway

The main limitation of standard attention for long sequences is the memory required to materialize and access the full attention matrix.