FTTransformerNetwork<T>

Models & Types · AiDotNet.NeuralNetworks.Tabular

FT-Transformer (Feature Tokenizer + Transformer) neural network for tabular data.

Inheritance: NeuralNetworkBase<T> → TabularNeuralNetworkBase<T> → FTTransformerNetwork<T>

For Beginners

FT-Transformer brings NLP’s Transformer to tables:

Architecture:

  1. Feature Tokenization: Each column becomes a “token” (like words in text)
  2. [CLS] Token: Special token to aggregate information for prediction
  3. Transformer Encoder: Self-attention learns feature relationships
  4. Classification Head: MLP on [CLS] token for final prediction

Key insight: Just as transformers revolutionized NLP by learning word relationships, FT-Transformer learns feature relationships through attention. Each feature can “attend” to other features to capture complex interactions.

This often outperforms gradient boosting on larger datasets and automatically learns which feature combinations matter.

How It Works

FT-Transformer applies the transformer architecture to tabular data by treating each feature as a token. It tokenizes numerical and categorical features into embeddings and processes them with standard transformer encoder layers. This implementation follows the standard neural network architecture pattern with:

  • Proper inheritance from NeuralNetworkBase
  • Layer-based architecture using ILayer components
  • Engine-based tensor operations for CPU/GPU acceleration
  • Full autodiff and JIT compilation support

Reference: “Revisiting Deep Learning Models for Tabular Data” (Gorishniy et al., NeurIPS 2021)

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var options = new FTTransformerOptions<double> { NumFeatures = 20, EmbeddingDimension = 192, NumHeads = 8, NumLayers = 3 };
var input = Tensor<float>.CreateRandom(new[] { 1, 20 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new FTTransformerNetwork<float>(architecture))
    .Build(trainX, trainY);
var output = result.Predict(input);

Constructors

Constructor Summary
FTTransformerNetwork() Initializes a new FT-Transformer network with the specified architecture.
FTTransformerNetwork(NeuralNetworkArchitecture<T> architecture, FTTransformerOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction)

Properties

Property Summary
FTTransformerOptions<T> Options { get; } Gets the FT-Transformer-specific options.
int EmbeddingDimension { get; } Gets the embedding dimension.
int NumHeads { get; } Gets the number of attention heads.
int NumLayers { get; } Gets the number of transformer layers.

Methods

Method Summary
Dictionary<string, T> GetFeatureImportance()
ModelMetadata<T> GetModelMetadata()