FTTransformerOptions<T>

Options & Configuration · AiDotNet.Models.Options

Configuration options for FT-Transformer, a Feature Tokenizer + Transformer for tabular data.

Inheritance: ModelOptions → NeuralNetworkOptions → ModelHyperparameterOptions → FinancialNeuralNetworkOptions → RiskModelOptions<T> → FTTransformerOptions<T>

For Beginners

FT-Transformer is a way to use the powerful Transformer architecture (the technology behind ChatGPT) on traditional tabular data like spreadsheets.

Key concepts:

  • Feature Tokenization: Each column in your data becomes a “token” (like words in text)
  • [CLS] Token: A special token added to aggregate information for the final prediction
  • Self-Attention: Allows the model to learn relationships between different features
  • No Manual Feature Engineering: The model learns which features interact automatically

Advantages:

  • Captures complex feature interactions through attention
  • Works well with both numerical and categorical features
  • Often outperforms gradient boosting on larger datasets
  • Provides attention weights for interpretability

Example usage:

How It Works

FT-Transformer applies the transformer architecture to tabular data by treating each feature as a token. It tokenizes numerical and categorical features into embeddings and processes them with standard transformer encoder layers.

Reference: “Revisiting Deep Learning Models for Tabular Data” (Gorishniy et al., NeurIPS 2021)

Constructors

Constructor Summary
FTTransformerOptions()

Properties

Property Summary
bool UseNumericalBias { get; set; } Gets or sets whether to use a bias term in the numerical feature embedding.
bool UsePreLayerNorm { get; set; } Gets or sets whether to use layer normalization before attention (Pre-LN) or after (Post-LN).
bool UseReGLU { get; set; } Gets or sets whether to use ReGLU activation in the feed-forward network.
double DropoutRate { get; set; } Gets or sets the dropout rate for attention and feed-forward layers.
double EmbeddingInitScale { get; set; } Gets or sets the initialization scale multiplier for feature-token embeddings (Gorishniy et al.
double LayerNormEpsilon { get; set; } Gets or sets the epsilon value for layer normalization numerical stability.
double WeightDecay { get; set; } Gets or sets the weight decay (L2 regularization) coefficient.
int EmbeddingDimension { get; set; } Gets or sets the embedding dimension for feature tokens.
int FeedForwardDimension { get; } Gets the feed-forward network dimension.
int FeedForwardMultiplier { get; set; } Gets or sets the feed-forward network dimension multiplier.
int NumHeads { get; set; } Gets or sets the number of attention heads.
int NumLayers { get; set; } Gets or sets the number of transformer encoder layers.
int[] CategoricalCardinalities { get; set; } Gets or sets the categorical feature cardinalities (number of unique values per categorical feature).

Methods

Method Summary
FTTransformerOptions<T> Clone() Creates a copy of the options.