TabTransformerNetwork<T>

Models & Types · AiDotNet.NeuralNetworks.Tabular

TabTransformer neural network for tabular data with categorical features.

Inheritance: NeuralNetworkBase<T> → TabularNeuralNetworkBase<T> → TabTransformerNetwork<T>

For Beginners

TabTransformer treats categorical features specially:

Architecture:

  1. Categorical Path: Embedding → Column Embedding → Transformer → Flatten
  2. Numerical Path: Pass through unchanged
  3. Concatenation: Combine both paths
  4. MLP Head: Final prediction layers

Key insight: Categorical features often have interactions that matter (e.g., “New York” + “Finance” vs “New York” + “Farming”). The transformer learns these relationships through self-attention.

Example flow: Categories [batch, num_cat] → Embeddings [batch, num_cat, embed_dim] → Transformer [batch, num_cat, embed_dim] → Flatten [batch, num_cat * embed_dim] ↘ Numericals [batch, num_num] → Concat [batch, num_cat * embed_dim + num_num] → MLP → Prediction

How It Works

TabTransformer applies transformer self-attention to categorical features while passing numerical features through directly. This captures complex relationships between categorical features that simple embeddings might miss. This implementation follows the standard neural network architecture pattern with:

  • Proper inheritance from NeuralNetworkBase
  • Layer-based architecture using ILayer components
  • Engine-based tensor operations for CPU/GPU acceleration
  • Full autodiff and JIT compilation support

Reference: “TabTransformer: Tabular Data Modeling Using Contextual Embeddings” (2020)

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var options = new TabTransformerOptions<double> { NumFeatures = 20, EmbeddingDimension = 32, NumHeads = 8, NumLayers = 6 };
var input = Tensor<float>.CreateRandom(new[] { 1, 20 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new TabTransformerNetwork<float>(architecture))
    .Build(trainX, trainY);
var output = result.Predict(input);

Constructors

Constructor Summary
TabTransformerNetwork() Initializes a new TabTransformer network with the specified architecture.
TabTransformerNetwork(NeuralNetworkArchitecture<T> architecture, TabTransformerOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction)

Properties

Property Summary
TabTransformerOptions<T> Options { get; } Gets the TabTransformer-specific options.
int EmbeddingDimension { get; } Gets the embedding dimension.
int NumHeads { get; } Gets the number of attention heads.
int NumLayers { get; } Gets the number of transformer layers.

Methods

Method Summary
Dictionary<string, T> GetFeatureImportance() Gets the learned feature importance from the attention layers.
ModelMetadata<T> GetModelMetadata()