FeatureTokenizer<T>
Models & Types · AiDotNet.NeuralNetworks.Tabular
Implements the Feature Tokenizer that converts tabular features into embeddings for FT-Transformer.
For Beginners
The Feature Tokenizer is like a translator that converts your spreadsheet columns into a format the Transformer can understand.
What it does:
- Takes each column value and converts it to a vector (list of numbers)
- For numbers (like age=25): Multiplies by learned weights and adds bias
- For categories (like color=“red”): Looks up a learned embedding vector
- Adds a special [CLS] token that will aggregate all feature information
Why this matters:
- Transformers work on sequences of vectors, not raw numbers
- This conversion allows the model to learn rich representations of each feature
- The [CLS] token provides a single representation for the final prediction
Example: If you have 5 features with embedding dimension 64: Input: [age, income, zip_code, gender, marital_status] Output: Tensor of shape [batch, 6, 64] (5 features + 1 CLS token, each as 64-dim vector)
How It Works
The Feature Tokenizer converts each input feature into a d-dimensional embedding:
- For numerical features: embedding = x * w + b (linear projection)
- For categorical features: embedding lookup from learned embedding table
A learnable [CLS] token is prepended to the sequence for classification/regression.
Constructors
| Constructor | Summary |
|---|---|
FeatureTokenizer(int numNumericalFeatures, int embeddingDimension, int[] categoricalCardinalities, bool useNumericalBias, double initScale) |
Initializes a new instance of the FeatureTokenizer class. |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
Gets the embedding dimension. |
int SequenceLength { get; } |
Gets the sequence length including [CLS] token. |
int TotalFeatures { get; } |
Gets the total number of features (numerical + categorical). |
long ParameterCount { get; } |
Gets the total number of trainable parameters. |
Methods
| Method | Summary |
|---|---|
Tensor<T> Forward(Tensor<T> numericalFeatures) |
Performs the forward pass with numerical features only. |
Tensor<T> Forward(Tensor<T> numericalFeatures, Matrix<int> categoricalIndices) |
Performs the forward pass to tokenize features into embeddings. |
Tensor<T> GetClsToken() |
Gets the [CLS] token embedding. |
Tensor<T> GetNumericalBias() |
Gets the numerical feature biases. |
Tensor<T> GetNumericalWeights() |
Gets the numerical feature weights. |
Vector<T> GetParameterGradients() |
Gets the parameter gradients as a single vector. |
Vector<T> GetParameters() |
Gets all trainable parameters as a single vector. |
void ResetGradients() |
Resets all gradients to zero. |
void SetParameters(Vector<T> parameters) |
Sets the trainable parameters from a vector. |
void UpdateParameters(T learningRate) |
Updates parameters using the calculated gradients. |