AttentionVariant
Enums · AiDotNet.Enums
Specifies the attention variant that determines how Key/Value heads map to Query heads.
For Beginners
In standard attention, each “head” has its own Query, Key, and Value.
Different variants change how many Key/Value heads are shared:
- MultiHead (MHA): Each head has its own K and V (standard, most memory)
- MultiQuery (MQA): All heads share one K and V (least memory, used by PaLM)
- GroupedQuery (GQA): Groups of heads share K and V (balanced, used by Llama 2/3)
GQA with numKVHeads=8 and numHeads=64 means each K/V head serves 8 Query heads, reducing KV-cache memory by 8x compared to standard MHA.
How It Works
Modern transformer architectures use different ratios of Query heads to Key/Value heads to trade off between model quality and memory efficiency during inference.
Fields
| Field | Summary |
|---|---|
AttentionVariant GroupedQuery |
Grouped-Query Attention where 1 < numKVHeads < numHeads (Ainslie et al., 2023). |
AttentionVariant MultiHead |
Standard Multi-Head Attention where numKVHeads == numHeads. |
AttentionVariant MultiQuery |
Multi-Query Attention where numKVHeads == 1 (Shazeer, 2019). |
int value__ |