FTTransformerOptions<T>
Options & Configuration · AiDotNet.Models.Options
Configuration options for FT-Transformer, a Feature Tokenizer + Transformer for tabular data.
Inheritance: ModelOptions → NeuralNetworkOptions → ModelHyperparameterOptions → FinancialNeuralNetworkOptions → RiskModelOptions<T> → FTTransformerOptions<T>
For Beginners
FT-Transformer is a way to use the powerful Transformer architecture (the technology behind ChatGPT) on traditional tabular data like spreadsheets.
Key concepts:
- Feature Tokenization: Each column in your data becomes a “token” (like words in text)
- [CLS] Token: A special token added to aggregate information for the final prediction
- Self-Attention: Allows the model to learn relationships between different features
- No Manual Feature Engineering: The model learns which features interact automatically
Advantages:
- Captures complex feature interactions through attention
- Works well with both numerical and categorical features
- Often outperforms gradient boosting on larger datasets
- Provides attention weights for interpretability
Example usage:
How It Works
FT-Transformer applies the transformer architecture to tabular data by treating each feature as a token. It tokenizes numerical and categorical features into embeddings and processes them with standard transformer encoder layers.
Reference: “Revisiting Deep Learning Models for Tabular Data” (Gorishniy et al., NeurIPS 2021)
Constructors
| Constructor | Summary |
|---|---|
FTTransformerOptions() |
Properties
| Property | Summary |
|---|---|
bool UseNumericalBias { get; set; } |
Gets or sets whether to use a bias term in the numerical feature embedding. |
bool UsePreLayerNorm { get; set; } |
Gets or sets whether to use layer normalization before attention (Pre-LN) or after (Post-LN). |
bool UseReGLU { get; set; } |
Gets or sets whether to use ReGLU activation in the feed-forward network. |
double DropoutRate { get; set; } |
Gets or sets the dropout rate for attention and feed-forward layers. |
double EmbeddingInitScale { get; set; } |
Gets or sets the initialization scale multiplier for feature-token embeddings (Gorishniy et al. |
double LayerNormEpsilon { get; set; } |
Gets or sets the epsilon value for layer normalization numerical stability. |
double WeightDecay { get; set; } |
Gets or sets the weight decay (L2 regularization) coefficient. |
int EmbeddingDimension { get; set; } |
Gets or sets the embedding dimension for feature tokens. |
int FeedForwardDimension { get; } |
Gets the feed-forward network dimension. |
int FeedForwardMultiplier { get; set; } |
Gets or sets the feed-forward network dimension multiplier. |
int NumHeads { get; set; } |
Gets or sets the number of attention heads. |
int NumLayers { get; set; } |
Gets or sets the number of transformer encoder layers. |
int[] CategoricalCardinalities { get; set; } |
Gets or sets the categorical feature cardinalities (number of unique values per categorical feature). |
Methods
| Method | Summary |
|---|---|
FTTransformerOptions<T> Clone() |
Creates a copy of the options. |