TabTransformerNetwork<T>
Models & Types · AiDotNet.NeuralNetworks.Tabular
TabTransformer neural network for tabular data with categorical features.
Inheritance: NeuralNetworkBase<T> → TabularNeuralNetworkBase<T> → TabTransformerNetwork<T>
For Beginners
TabTransformer treats categorical features specially:
Architecture:
- Categorical Path: Embedding → Column Embedding → Transformer → Flatten
- Numerical Path: Pass through unchanged
- Concatenation: Combine both paths
- MLP Head: Final prediction layers
Key insight: Categorical features often have interactions that matter (e.g., “New York” + “Finance” vs “New York” + “Farming”). The transformer learns these relationships through self-attention.
Example flow: Categories [batch, num_cat] → Embeddings [batch, num_cat, embed_dim] → Transformer [batch, num_cat, embed_dim] → Flatten [batch, num_cat * embed_dim] ↘ Numericals [batch, num_num] → Concat [batch, num_cat * embed_dim + num_num] → MLP → Prediction
How It Works
TabTransformer applies transformer self-attention to categorical features while passing numerical features through directly. This captures complex relationships between categorical features that simple embeddings might miss. This implementation follows the standard neural network architecture pattern with:
- Proper inheritance from NeuralNetworkBase
- Layer-based architecture using ILayer components
- Engine-based tensor operations for CPU/GPU acceleration
- Full autodiff and JIT compilation support
Reference: “TabTransformer: Tabular Data Modeling Using Contextual Embeddings” (2020)
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var options = new TabTransformerOptions<double> { NumFeatures = 20, EmbeddingDimension = 32, NumHeads = 8, NumLayers = 6 };
var input = Tensor<float>.CreateRandom(new[] { 1, 20 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new TabTransformerNetwork<float>(architecture))
.Build(trainX, trainY);
var output = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
TabTransformerNetwork() |
Initializes a new TabTransformer network with the specified architecture. |
TabTransformerNetwork(NeuralNetworkArchitecture<T> architecture, TabTransformerOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Properties
| Property | Summary |
|---|---|
TabTransformerOptions<T> Options { get; } |
Gets the TabTransformer-specific options. |
int EmbeddingDimension { get; } |
Gets the embedding dimension. |
int NumHeads { get; } |
Gets the number of attention heads. |
int NumLayers { get; } |
Gets the number of transformer layers. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetFeatureImportance() |
Gets the learned feature importance from the attention layers. |
ModelMetadata<T> GetModelMetadata() |