Multi-head attention

Summary

Aggregates different perspectives

This approach allows the model to attend to different parts of the input sequence simultaneously, capturing various aspects of the context more effectively.

Like a hydra, it focuses on different aspects of the context. Getting a finer understanding.

Multi-head attention captures more context by dividing the input processing into multiple independent attention heads. Each head focuses on different parts of the input and captures diverse types of relationships, both local and global. This parallelism allows the model to learn multiple perspectives simultaneously, enriching its understanding of the input sequence and improving performance on complex tasks like language modeling, machine translation, and more.

Related to the Attention mechanism.

Multi-Head Attention

In multi-head attention, the idea is to split the input into multiple subsets of attention heads. Instead of computing a single attention score for each token pair, multiple attention “heads” are used, with each head attending to different parts of the input. This provides several benefits:

a) Diverse Attention Patterns

Each head in multi-head attention can focus on different aspects of the input sequence, allowing the model to capture multiple relationships between tokens. For instance:

• One head may focus on syntactic relationships (like word order or structure).
• Another head may focus on semantic relationships (like the meaning or context of words).

By having multiple attention heads, the model can learn to capture different types of context simultaneously.

b) Different Projection Spaces

Each attention head has its own set of parameters, which project the input into a different subspace (i.e., they use different weight matrices to transform queries, keys, and values). This allows each head to learn different relationships between tokens in various representational subspaces, thus increasing the model’s capacity to understand complex dependencies.

For example, one head might focus on short-range dependencies (like adjacent words), while another head could capture long-range dependencies (like relationships between words at opposite ends of the sentence).

c) Improved Expressiveness

By combining the outputs of multiple heads, the model gains a richer representation of the context. Each attention head contributes unique insights about how tokens in the sequence relate to each other, and by concatenating these different perspectives, the overall attention mechanism becomes more expressive.

Multi-Head Attention Process

The process for multi-head attention involves the following steps:

1. Linear Transformations: The input vectors (representing words or tokens) are linearly transformed into queries (Q), keys (K), and values (V) for each head. These transformations are different for each head, allowing each head to capture different relationships in the data.

2. Attention Calculation for Each Head: For each head, scaled dot-product attention is calculated independently. The attention mechanism computes scores by comparing queries and keys, and these scores are used to weigh the values. This results in a unique context vector for each head.

3. Concatenation: The outputs from all the attention heads are concatenated into a single vector. This step combines the different perspectives learned by each head.

4. Final Linear Transformation: After concatenation, a final linear transformation is applied to combine the information from all heads into a single vector that can be used in the next layer of the model.

How Multi-Head Attention Captures More Context

Multi-head attention captures more context than single-head attention for several reasons:

• ==Multiple Focus Areas: By using multiple heads, the model can simultaneously focus on different parts of the sequence. Some heads might attend to local dependencies, while others might capture more distant relationships. This gives the model a broader understanding== of the entire sequence.

• Handling Ambiguity: In natural language, the meaning of words can depend heavily on context. Multi-head attention allows the model to disambiguate meanings by attending to different context clues in parallel. For instance, the word “bank” in “I went to the bank” can mean different things, and different heads can capture clues from the surrounding words to determine whether “bank” refers to a financial institution or a riverbank.

• Diverse Representations: Each head transforms the input into different representational subspaces, meaning the model learns diverse representations of the same input. This diversity enhances the model’s ability to generalize and capture complex relationships in the data.

Application Example: Language Translation

Consider translating a sentence from one language to another. The multi-head attention mechanism in the Transformer model helps capture different linguistic structures:

• One head might focus on aligning subject-verb pairs between the two languages.
• Another head might capture longer dependencies, like how nouns and pronouns refer to each other across a long sentence.
• A third head might capture grammatical structure differences between the source and target languages.

By aggregating these different perspectives, multi-head attention ensures that the model understands both the local and global context, leading to better translation quality.