AttentionVariant

Enums · AiDotNet.Enums

Specifies the attention variant that determines how Key/Value heads map to Query heads.

For Beginners

In standard attention, each “head” has its own Query, Key, and Value.

Different variants change how many Key/Value heads are shared:

  • MultiHead (MHA): Each head has its own K and V (standard, most memory)
  • MultiQuery (MQA): All heads share one K and V (least memory, used by PaLM)
  • GroupedQuery (GQA): Groups of heads share K and V (balanced, used by Llama 2/3)

GQA with numKVHeads=8 and numHeads=64 means each K/V head serves 8 Query heads, reducing KV-cache memory by 8x compared to standard MHA.

How It Works

Modern transformer architectures use different ratios of Query heads to Key/Value heads to trade off between model quality and memory efficiency during inference.

Fields

Field Summary
AttentionVariant GroupedQuery Grouped-Query Attention where 1 < numKVHeads < numHeads (Ainslie et al., 2023).
AttentionVariant MultiHead Standard Multi-Head Attention where numKVHeads == numHeads.
AttentionVariant MultiQuery Multi-Query Attention where numKVHeads == 1 (Shazeer, 2019).
int value__