Achieving High Performance and Efficiency in LLMs by Grouped Query Attention

One of the most important tasks for an LLM is to be able to attend to different parts of the input text when answering a question. This is because the answer to a question may be located in different parts of the text, and the LLM needs to be able to identify and focus on the relevant information.

Let's get into more details. The rest of the article is categorized in below sections.

• Introduction
• Problem
• Solution
• How GQA Works?
• Examples
• Comparison
• Conclusion

Introduction

Large language models (LLMs) are a type of artificial intelligence (AI) that are trained on massive datasets of text and code. They can be used for a variety of tasks, such as machine translation, question answering, and text summarization.

Problem

One of the challenges of LLMs is that they can be computationally expensive to train and run. This is because they typically have a large number of parameters, which need to be updated during training.

One way to approach above mentioned problem is to use attention mechanisms. Attention mechanisms allow the LLM to learn how to focus on different parts of the input text, based on the question that is being asked.

There are two main types of attention mechanisms that are used in LLMs: multi-head attention and multi-query attention.

Multi-head attention uses multiple attention heads to attend to different parts of the input text. This allows the LLM to learn more complex relationships between the different parts of the text.

Multi-query attention allows the LLM to attend to multiple queries at the same time. This is useful for tasks such as question answering, where the LLM needs to answer multiple questions about the same piece of text.

Solution

Grouped query attention (GQA) is a new attention mechanism that is designed to achieve both high performance and efficiency in LLMs. GQA is an interpolation between multi-head attention and multi-query attention.

It uses multiple attention heads, but each group of attention heads shares the same key and value vectors. This allows GQA to achieve the performance of multi-head attention, while also being more efficient.

How GQA Works

Grouped-query attention (GQA) is a technique for improving the efficiency of multi-head attention (MHA), a key component of modern language models (LLMs) like BERT and GPT-3.

MHA works by dividing the input sequence into three sets of vectors: queries, keys, and values. Each query vector is then compared to all of the key vectors, and a weight is assigned to each key vector based on how similar it is to the query vector. The values associated with the key vectors are then weighted and summed to produce an output vector.

This process is repeated for each query vector, resulting in a set of output vectors that represent the attention-weighted average of the input sequence.

MHA is very effective at modeling long-range dependencies in text, but it can be computationally expensive, especially for large LLMs.

GQA works by grouping the query vectors into smaller groups and sharing the key and value vectors across each group. This reduces the number of computations required to perform MHA, resulting in significant speedups.

Here is a simple implementation of GQA in Python:

import torch
from torch import nn

class GroupedQueryAttention(nn.Module):
    def __init__(self, num_heads, group_size):
        super().__init__()

        self.num_heads = num_heads
        self.group_size = group_size

        self.query_proj = nn.Linear(d_model, num_heads * d_k)
        self.key_proj = nn.Linear(d_model, num_heads * d_k)
        self.value_proj = nn.Linear(d_model, num_heads * d_v)

        self.out_proj = nn.Linear(num_heads * d_v, d_model)

    def forward(self, queries, keys, values):
        """
        Args:
            queries: (batch_size, seq_len, d_model)
            keys: (batch_size, seq_len, d_model)
            values: (batch_size, seq_len, d_model)

        Returns:
            output: (batch_size, seq_len, d_model)
        """

        # Project queries, keys, and values to the desired dimension.
        queries = self.query_proj(queries)
        keys = self.key_proj(keys)
        values = self.value_proj(values)

        # Group the query vectors.
        grouped_queries = queries.view(queries.size(0), queries.size(1) // self.group_size, self.group_size, -1)

        # Share the key and value vectors across each group.
        grouped_keys = keys.unsqueeze(1).expand(grouped_queries.size())
        grouped_values = values.unsqueeze(1).expand(grouped_queries.size())

        # Perform multi-head attention on each group.
        attentions = torch.matmul(grouped_queries, grouped_keys.transpose(-1, -2))
        attentions = attentions.softmax(-1)
        grouped_output = torch.matmul(attentions, grouped_values)

        # Flatten the grouped output.
        output = grouped_output.view(grouped_queries.size(0), -1, grouped_output.size(-1))

        # Project the output to the desired dimension.
        output = self.out_proj(output)

        return output

# Example usage:

grouped_query_attention = GroupedQueryAttention(num_heads=8, group_size=2)

queries = torch.randn(batch_size, seq_len, d_model)
keys = torch.randn(batch_size, seq_len, d_model)
values = torch.randn(batch_size, seq_len, d_model)

output = grouped_query_attention(queries, keys, values)

Examples

Here is an example of how GQA can be used to answer a question. Let's say the question is What is the capital of France?

The query vector for this question would be a vector that represents the words "what" and "capital".

The key vectors would be vectors that represent each word in the input text, which in this case is the sentence Paris is the capital of France. The value vectors would be vectors that represent the meaning of each word in the input text.

GQA would then compute three attention scores, one for each attention head. The attention scores would be computed by taking the dot product of the query vector and the key vectors. The attention scores would then be used to compute a weighted sum of the value vectors. The weighted sum would be the output of the GQA layer.

The output of the GQA layer would be a vector that represents the answer to the question. In this case, the output vector would be a vector that represents the word "Paris".

Comparison with Other Available Solutions

Grouped query attention is a relatively new technique, and there are not many other solutions that are directly comparable. However, it can be compared to multi-head attention and multi-query attention.

Multi-head attention is a more traditional approach to attention. It uses multiple attention heads to attend to different parts of the input. This can improve the accuracy of the model, but it can also make the model more computationally expensive.

Multi-query attention is a simpler approach to attention. It uses a single attention head to attend to multiple queries. This can improve the efficiency of the model, but it can also make the model less accurate.

Grouped query attention combines the benefits of multi-head attention and multi-query attention. It is more efficient than multi-head attention, and it is more accurate than multi-query attention.

Conclusion

GQA has been shown to achieve comparable performance to MHA on a variety of NLP tasks, while being significantly more efficient. For example, on the GLUE benchmark, GQA achieves 91.4% accuracy, which is only slightly lower than the 91.6% accuracy achieved by MHA. However, GQA is up to 3x faster than MHA.

I hope this article has given you a good understanding of GQA. If you enjoyed the article do consider checking more.