1.6.6
Flash Attention 4: Blackwell Architecture
What is it?
FlashAttention 4 targets NVIDIA Blackwell architecture GPUs such as B200 and GB200. The book describes how faster Tensor Core matrix operations shift the performance bottleneck toward softmax exponentials and shared-memory traffic. FlashAttention 4 addresses this through asynchronous computation pipelines, software-emulated exponentials, conditional softmax rescaling and new backward-pass techniques.
Key points
1. FlashAttention 4 targets NVIDIA Blackwell GPUs.
2. Fully asynchronous MMA pipelines overlap matrix operations and data movement.
3. TMA loads can be coordinated with computation.
4. Software-emulated exponentials use polynomial approximations executed on Tensor Cores.
5. Conditional softmax rescaling avoids unnecessary updates when the new tile maximum does not exceed the running maximum.
6. The backward pass uses Blackwell Tensor Memory and 2-CTA cooperative execution.
7. CuTe-DSL is used for the FlashAttention 4 implementation.
8. The implementation is designed around Blackwell hardware characteristics.
Example: Why FlashAttention 4 changes the pipeline
Imagine a GPU where matrix multiplication becomes much faster, but softmax exponential operations do not improve at the same rate.
The matrix multiplication unit may finish its work quickly and then wait for other operations.
FlashAttention 4 addresses this imbalance by:
1. Overlapping asynchronous matrix operations and data transfers.
2. Emulating exponential calculations using Tensor Core operations.
3. Skipping unnecessary softmax rescaling when possible.
4. Improving backward-pass memory coordination.
The objective is to keep more of the GPU hardware productively occupied.
Real-world application
FlashAttention 4 is relevant to engineers optimizing attention kernels for Blackwell GPU systems and studying the relationship between algorithms and hardware architecture.
Key takeaway
FlashAttention 4 adapts attention computation to Blackwell hardware by optimizing asynchronous execution, exponential calculations and memory coordination.