CLSToken<T>
Models & Types · AiDotNet.NeuralNetworks.Tabular
CLS (Classification) Token for transformer-based tabular models.
For Beginners
The CLS token serves as a “summary” position:
- It’s added to the beginning of your features
- The transformer allows it to attend to all features
- After processing, the CLS token “knows about” all features
- We use its final representation for prediction
This is the same approach used in BERT for text classification.
How It Works
The CLS token is a learnable embedding prepended to the input sequence. After transformer processing, the CLS token’s representation is used as an aggregate representation of the entire input for classification/regression.
Constructors
| Constructor | Summary |
|---|---|
CLSToken(int embeddingDimension, double initScale) |
Initializes a new CLS token with the specified embedding dimension. |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
Gets the embedding dimension. |
long ParameterCount { get; } |
Gets the number of parameters. |
Methods
| Method | Summary |
|---|---|
Tensor<T> ExtractCLS(Tensor<T> transformerOutput) |
Extracts the CLS token representation from transformer output. |
Tensor<T> GetEmbedding() |
Gets the current CLS embedding values. |
Tensor<T> PrependCLS(Tensor<T> embeddings) |
Prepends the CLS token to the input embeddings. |
void ResetGradients() |
Resets gradients to zero. |
void UpdateParameters(T learningRate) |
Updates the CLS token embedding. |