AttentionKernelPath

Enums · AiDotNet.Enums

Identifies which attention kernel a layer’s most recent forward pass actually executed.

For Beginners

Attention can be computed two ways that give the same answer. One of them also hands back a big table of “how much each token looked at each other token”; that table is useful for diagnostics but costs a lot of memory on long sequences. This enum records which of the two was used last.

How It Works

The same attention block can be computed by more than one kernel, and the choice is a memory decision rather than a mathematical one: the fused FlashAttention kernel never returns the [batch, heads, seq_q, seq_kv] score matrix, so a forward that needs those scores (the auxiliary losses, or a manual tape-free backward) has to take the score-materializing kernel instead.

Reporting the path a forward took makes that decision observable, so a regression that silently reverts a layer to the expensive kernel is a failing assertion rather than a quiet slowdown (#2312).

Fields

Field Summary
AttentionKernelPath AdditiveBias The additive-bias path used when a layer injects an ALiBi positional bias.
AttentionKernelPath FusedFlashAttention The fused FlashAttention kernel (IEngine.FlashAttention), which keeps one log-sum-exp per query row instead of the full score matrix and masks causally in-kernel.
AttentionKernelPath FusedMultiHeadCore The head-interleaved fused kernel (IEngine.MultiHeadAttentionCore), which reads the projected Q/K/V in their [batch, seq, heads*headDim] layout and returns the merged-head context, so the head-split and head-merge permutes are part of t…
AttentionKernelPath None No attention forward has run yet on this layer.
AttentionKernelPath ScoreMaterializing The score-materializing kernel (IEngine.ScaledDotProductAttention), which returns the attention weights because something downstream reads them.
int value__