FTTransformerNetwork<T>
Models & Types · AiDotNet.NeuralNetworks.Tabular
FT-Transformer (Feature Tokenizer + Transformer) neural network for tabular data.
Inheritance: NeuralNetworkBase<T> → TabularNeuralNetworkBase<T> → FTTransformerNetwork<T>
For Beginners
FT-Transformer brings NLP’s Transformer to tables:
Architecture:
- Feature Tokenization: Each column becomes a “token” (like words in text)
- [CLS] Token: Special token to aggregate information for prediction
- Transformer Encoder: Self-attention learns feature relationships
- Classification Head: MLP on [CLS] token for final prediction
Key insight: Just as transformers revolutionized NLP by learning word relationships, FT-Transformer learns feature relationships through attention. Each feature can “attend” to other features to capture complex interactions.
This often outperforms gradient boosting on larger datasets and automatically learns which feature combinations matter.
How It Works
FT-Transformer applies the transformer architecture to tabular data by treating each feature as a token. It tokenizes numerical and categorical features into embeddings and processes them with standard transformer encoder layers. This implementation follows the standard neural network architecture pattern with:
- Proper inheritance from NeuralNetworkBase
- Layer-based architecture using ILayer components
- Engine-based tensor operations for CPU/GPU acceleration
- Full autodiff and JIT compilation support
Reference: “Revisiting Deep Learning Models for Tabular Data” (Gorishniy et al., NeurIPS 2021)
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var options = new FTTransformerOptions<double> { NumFeatures = 20, EmbeddingDimension = 192, NumHeads = 8, NumLayers = 3 };
var input = Tensor<float>.CreateRandom(new[] { 1, 20 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new FTTransformerNetwork<float>(architecture))
.Build(trainX, trainY);
var output = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
FTTransformerNetwork() |
Initializes a new FT-Transformer network with the specified architecture. |
FTTransformerNetwork(NeuralNetworkArchitecture<T> architecture, FTTransformerOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Properties
| Property | Summary |
|---|---|
FTTransformerOptions<T> Options { get; } |
Gets the FT-Transformer-specific options. |
int EmbeddingDimension { get; } |
Gets the embedding dimension. |
int NumHeads { get; } |
Gets the number of attention heads. |
int NumLayers { get; } |
Gets the number of transformer layers. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetFeatureImportance() |
|
ModelMetadata<T> GetModelMetadata() |