FTTransformerBase<T>
Base Classes · AiDotNet.NeuralNetworks.Tabular
Base implementation of FT-Transformer (Feature Tokenizer + Transformer) for tabular data.
Implements: IParameterSource<T>
Known derived types: FTTransformerClassifier<T>, FTTransformerRegression<T>
For Beginners
FT-Transformer brings the power of the Transformer architecture (used in GPT, BERT, etc.) to traditional tabular data like spreadsheets and databases.
Architecture overview:
- Feature Tokenizer: Converts each column value into a vector embedding
- [CLS] Token: A special learnable token prepended to capture global information
- Transformer Layers: Self-attention layers to capture feature interactions
- Prediction Head: Uses the [CLS] token output to make predictions
How it works:
- Each feature (column) becomes a “token” with its own embedding
- Self-attention allows any feature to “look at” any other feature
- The [CLS] token aggregates information from all features
- Final prediction is made based on the [CLS] representation
Key advantages:
- Learns which features interact with each other automatically
- No manual feature engineering needed
- Often outperforms gradient boosting on larger datasets
- Attention weights provide interpretability
How It Works
FT-Transformer is a simple but effective adaptation of the Transformer architecture for tabular data. It treats each feature as a token by embedding it into a d-dimensional vector, then processes the sequence with standard transformer encoder layers.
Reference: “Revisiting Deep Learning Models for Tabular Data” (Gorishniy et al., NeurIPS 2021)
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
Gets the embedding dimension. |
int NumLayers { get; } |
Gets the number of transformer layers. |
int SequenceLength { get; } |
Gets the sequence length including [CLS] token. |
long ParameterCount { get; } |
Methods
| Method | Summary |
|---|---|
List<Tensor<T>> GetAttentionWeights() |
Gets attention weights from all layers for interpretability. |
Vector<T> GetFeatureImportance() |
Computes feature importance based on the attention patterns. |
Vector<T> GetParameterGradients() |
Gets parameter gradients as a single vector. |
Vector<T> GetParameters() |
Gets all trainable parameters as a single vector. |
void ResetState() |
Resets internal state including caches and gradients. |
void SetParameters(Vector<T> parameters) |
Sets all trainable parameters from a vector. |
void SetTrainingMode(bool isTraining) |
Sets the training mode for the model. |
void UpdateParameters(T learningRate) |
Updates all parameters using the calculated gradients. |