CLSToken<T>

Models & Types · AiDotNet.NeuralNetworks.Tabular

CLS (Classification) Token for transformer-based tabular models.

For Beginners

The CLS token serves as a “summary” position:

  • It’s added to the beginning of your features
  • The transformer allows it to attend to all features
  • After processing, the CLS token “knows about” all features
  • We use its final representation for prediction

This is the same approach used in BERT for text classification.

How It Works

The CLS token is a learnable embedding prepended to the input sequence. After transformer processing, the CLS token’s representation is used as an aggregate representation of the entire input for classification/regression.

Constructors

Constructor Summary
CLSToken(int embeddingDimension, double initScale) Initializes a new CLS token with the specified embedding dimension.

Properties

Property Summary
int EmbeddingDimension { get; } Gets the embedding dimension.
long ParameterCount { get; } Gets the number of parameters.

Methods

Method Summary
Tensor<T> ExtractCLS(Tensor<T> transformerOutput) Extracts the CLS token representation from transformer output.
Tensor<T> GetEmbedding() Gets the current CLS embedding values.
Tensor<T> PrependCLS(Tensor<T> embeddings) Prepends the CLS token to the input embeddings.
void ResetGradients() Resets gradients to zero.
void UpdateParameters(T learningRate) Updates the CLS token embedding.