PrependCLSTokenLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Prepends a learnable [CLS] token to a sequence-of-embeddings input, as introduced by BERT (Devlin et al. 2018 §3.1) and adopted by ViT / AST (Dosovitskiy et al. 2020 §3.1; Gong et al. 2021 §2.2). The CLS token starts as a learnable parameter [1, embedDim]; the layer broadcasts it across the batch and concatenates it at sequence position 0 so the transformer’s first output position becomes the classification representation.

Inheritance: LayerBase<T> → PrependCLSTokenLayer<T>

Implements: IShapeContract

For Beginners

Most transformer classifiers prepend a special learnable token to the input sequence; the network learns to use that one token as a “summary slot” for the whole sequence. After the transformer runs, you read just that one position to get the classification feature — no mean-pooling required, and gradient flow during training teaches the CLS token to aggregate task-relevant information from the rest of the sequence.

How It Works

Pairs with SequenceTokenSliceLayer using First after the transformer stack to extract the trained classification embedding — the canonical AST / ViT classification head.

Constructors

Constructor Summary
PrependCLSTokenLayer(int embedDim, double initScale, int? seed) Creates a CLS-token prepender for embedDim-wide inputs.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).