FTTransformerBase<T>

Base Classes · AiDotNet.NeuralNetworks.Tabular

Base implementation of FT-Transformer (Feature Tokenizer + Transformer) for tabular data.

Implements: IParameterSource<T>

Known derived types: FTTransformerClassifier<T>, FTTransformerRegression<T>

For Beginners

FT-Transformer brings the power of the Transformer architecture (used in GPT, BERT, etc.) to traditional tabular data like spreadsheets and databases.

Architecture overview:

  1. Feature Tokenizer: Converts each column value into a vector embedding
  2. [CLS] Token: A special learnable token prepended to capture global information
  3. Transformer Layers: Self-attention layers to capture feature interactions
  4. Prediction Head: Uses the [CLS] token output to make predictions

How it works:

  • Each feature (column) becomes a “token” with its own embedding
  • Self-attention allows any feature to “look at” any other feature
  • The [CLS] token aggregates information from all features
  • Final prediction is made based on the [CLS] representation

Key advantages:

  • Learns which features interact with each other automatically
  • No manual feature engineering needed
  • Often outperforms gradient boosting on larger datasets
  • Attention weights provide interpretability

How It Works

FT-Transformer is a simple but effective adaptation of the Transformer architecture for tabular data. It treats each feature as a token by embedding it into a d-dimensional vector, then processes the sequence with standard transformer encoder layers.

Reference: “Revisiting Deep Learning Models for Tabular Data” (Gorishniy et al., NeurIPS 2021)

Properties

Property Summary
int EmbeddingDimension { get; } Gets the embedding dimension.
int NumLayers { get; } Gets the number of transformer layers.
int SequenceLength { get; } Gets the sequence length including [CLS] token.
long ParameterCount { get; }

Methods

Method Summary
List<Tensor<T>> GetAttentionWeights() Gets attention weights from all layers for interpretability.
Vector<T> GetFeatureImportance() Computes feature importance based on the attention patterns.
Vector<T> GetParameterGradients() Gets parameter gradients as a single vector.
Vector<T> GetParameters() Gets all trainable parameters as a single vector.
void ResetState() Resets internal state including caches and gradients.
void SetParameters(Vector<T> parameters) Sets all trainable parameters from a vector.
void SetTrainingMode(bool isTraining) Sets the training mode for the model.
void UpdateParameters(T learningRate) Updates all parameters using the calculated gradients.