FlashAttention-3 unleashes the facility of H100 GPUs for LLMs

Consideration is a core part of the transformer structure utilized in massive language fashions (LLMs). However as LLMs develop bigger and deal with longer enter sequences, the computational price of consideration turns into a bottleneck. 

To handle this problem, researchers from Colfax Analysis, Meta, Nvidia, Georgia Tech, Princeton College, and Collectively AI have launched FlashAttention-3, a brand new approach that considerably quickens consideration computation on Nvidia Hopper GPUs (H100 and H800).

FlashAttention-3 builds upon earlier work on FlashAttention and FlashAttention-2 and additional optimizes using assets on Nvidia Hopper GPUs to maximise efficiency and effectivity for LLM coaching and inference. 

The problem of consideration computation in LLMs

One of many key improvements of transformers is the eye mechanism, which permits the mannequin to compute the connection between totally different tokens in an enter sequence.

Whereas the eye mechanism may be very efficient, it is usually computationally costly. The price of consideration computation grows quadratically with the size of the enter sequence. As LLMs are scaled to deal with longer and longer enter sequences, the eye mechanism turns into a serious bottleneck. 

Moreover, trendy {hardware} accelerators corresponding to GPUs are optimized for matrix multiplication (matmul) operations, that are the constructing blocks of deep studying fashions. These accelerators even have computational models for different forms of operations corresponding to exponentiation, however these models are tons of of instances slower than the matmul parts.

Consideration computations use a mix of matrix multiplications and different particular capabilities that aren’t as optimized for GPUs.

For instance, the softmax perform, which is used to normalize the eye weights, is computationally costlier than matrix multiplication. Because of this, though matrix multiplications account for many of the computations in consideration, the general computation can get slowed down by a small variety of particular capabilities.

One of many essential features of optimizing consideration computation is to schedule the workloads in a means that operations don’t get blocked by one another and make environment friendly use of various kinds of reminiscence parts. 

Making higher use of {hardware} assets

FlashAttention, launched in 2022, addressed the challenges of computing consideration by decreasing the variety of reminiscence reads and writes between GPU excessive bandwidth reminiscence (HBM) and GPU on-chip static random entry reminiscence (SRAM) when doing consideration computation. As an alternative of computing the eye weights for the whole sequence directly, FlashAttention breaks down the computation into smaller chunks, known as “tiles,” that may be processed extra effectively on GPUs.

FlashAttention has been extensively adopted and has contributed to growing the context window of LLMs from just a few thousand tokens to tons of of hundreds and even thousands and thousands of tokens. 

Nonetheless, as {hardware} has improved, so have the chances of optimizing LLM computations. FlashAttention-2, launched in 2023, additional optimized using GPU assets, attaining as much as 70% of the declared most efficiency on Nvidia A100 GPUs. Nonetheless, the identical optimizations didn’t switch to the newer H100 GPUs. FlashAttention-2 solely used 35% of H100’s most capability.

FlashAttention-3

FlashAttention-3 takes benefit of recent options in Nvidia Hopper GPUs to maximise efficiency. These options allow larger throughput on matrix multiplication operations, quicker information switch throughout totally different reminiscence segments, and higher effectivity on low-precision operations.

FlashAttention-3 introduces a number of improvements to enhance the efficiency of consideration computation on H100 GPUs.

FlashAttention-3 schedules operations in a means that maximizes the overlap between computation and the motion of knowledge between totally different reminiscence segments of the GPU. This reduces the time the GPU spends idle ready for information to be transferred. It additionally interleaves the matrix multiplication and softmax operations to scale back the potential of bottlenecks in computing consideration values.

FlashAttention-3 additionally makes use of a particular association of operations for quicker and extra correct computations of consideration in quantized fashions. Quantization is a well-liked approach that reduces the scale of fashions by utilizing low-bit numbers to retailer their weights. The tradeoff of quantization is the potential lack of accuracy. FlashAttention-3 addresses this drawback by fastidiously arranging the computations to attenuate the affect of quantization on accuracy.

Based on the researchers, FlashAttention-3 achieves as much as 75% utilization of the H100 GPU’s most capabilities. This interprets to a 1.5–2x speedup in comparison with earlier variations of FlashAttention for each coaching and operating LLMs.

The advantages of FlashAttention-3

The quicker consideration computation supplied by FlashAttention-3 has a number of implications for LLM improvement and purposes.

Coaching LLMs is a computationally costly course of that may take weeks and even months. The quick consideration computation supplied by FlashAttention-3 can considerably cut back the time it takes to coach LLMs, which may allow researchers and builders to experiment with bigger fashions and datasets.

FlashAttention-3 can even assist lengthen the context window of LLMs by enabling them to course of longer sequences extra effectively. This could unlock new purposes for LLMs in areas corresponding to long-form doc understanding and many-shot in-context studying.

And by utilizing a better share of GPU capability, FlashAttention-3 can cut back the variety of accelerators required to run LLMs and slash the price of operating fashions in manufacturing.

The researchers have open-sourced FlashAttention-3 beneath a permissive license and plan to combine it into in style deep studying libraries corresponding to PyTorch and Hugging Face Transformers. It will make it simpler for researchers and builders to benefit from the efficiency advantages of FlashAttention-3.
“We’ve seen that designing algorithms that benefit from the {hardware} they run on can carry vital effectivity good points and unlock new mannequin capabilities corresponding to lengthy context,” the researchers wrote in a weblog publish printed by Collectively AI. “We look ahead to future work on optimization for LLM inference, in addition to generalizing our methods to different {hardware} architectures.”