SequencePoolingMode

Enums · AiDotNet.Enums

Strategy for collapsing a transformer encoder’s [batch, seq, dim] hidden states into a single [batch, dim] vector before the classification head, when the task is single-label per sequence.

How It Works

Picking the wrong mode silently destroys the position-specific signal the model needs to learn. The default LastToken matches canonical autoregressive language modelling (GPT-style next-token prediction): the last position has attended to every preceding position via causal self-attention and is therefore the natural summary of the prefix.

MeanPool averages over all positions and is appropriate for non-causal sequence-classification (e.g. document sentiment where the whole sequence is observed at once and word order matters less than the overall content). Using MeanPool for next-token LM produced the flat-softmax convergence failure tracked in AiDotNet#1232: every context mapped to roughly the same averaged hidden state, the model couldn’t learn position-conditioned outputs, and softmax converged to ~uniform / V.

ClsToken matches BERT-style models that prepend a dedicated [CLS] token and use its final-layer hidden state as the sequence summary. None keeps the full [batch, seq, dim] shape — the right choice when the loss is applied per-token (token-classification, masked-LM with parallel position prediction, sequence-to-sequence training).

Fields

Field Summary
SequencePoolingMode ClsToken Use the FIRST position’s hidden state, treated as a prepended [CLS] summary token (BERT-style).
SequencePoolingMode LastToken Take the LAST position’s hidden state.
SequencePoolingMode MeanPool Average all positions.
SequencePoolingMode None Skip pooling entirely.
int value__