1.6.4
Flash Attention 2: Better Parallelism
What is it?
FlashAttention 2 improves the original FlashAttention implementation through better work partitioning and GPU parallelism. It reduces unnecessary non-matrix-multiplication operations, parallelizes across the query sequence dimension and skips fully masked blocks in causal attention. These changes improve hardware utilization and reduce execution overhead.
Key points
1. FlashAttention 2 reduces unnecessary non-matmul operations.
2. It improves work partitioning across GPU warps.
3. It adds parallelism across the query sequence dimension.
4. This helps GPU utilization for long sequences and small batch sizes.
5. It optimizes causal masking.
6. Fully masked blocks can be skipped.
7. Skipping masked blocks reduces unnecessary computation for causal attention.
Example: Skipping masked attention blocks
In causal attention, a token cannot attend to future tokens.
For a sequence of four positions:
Position 1 can attend to: 1
Position 2 can attend to: 1, 2
Position 3 can attend to: 1, 2, 3
Position 4 can attend to: 1, 2, 3, 4
The upper triangular region of the attention matrix is fully masked.
FlashAttention 2 can skip fully masked blocks rather than spending time processing them.
Real-world application
FlashAttention 2 is useful for improving attention performance on GPUs, especially for long sequences and autoregressive workloads.
Key takeaway
FlashAttention 2 improves parallelism, reduces non-matmul overhead and avoids unnecessary work on fully masked causal blocks.