SAINTNetwork<T>

Models & Types · AiDotNet.NeuralNetworks.Tabular

SAINT (Self-Attention and Intersample Attention Transformer) neural network for tabular data.

Inheritance: NeuralNetworkBase<T> → TabularNeuralNetworkBase<T> → SAINTNetwork<T>

For Beginners

SAINT is powerful because it learns two things:

Architecture:

  1. Column Attention: Which features are related to each other?
  • Self-attention within each sample across features
  1. Row Attention: Which training samples are similar?
  • Inter-sample attention comparing samples in the batch
  1. Alternating Layers: Column and row attention alternate through the network
  2. MLP Head: Final prediction layers

Key insight: By comparing samples within a batch, SAINT can leverage patterns that similar samples share, making it especially effective for semi-supervised learning and when samples have meaningful relationships.

Example flow: Features [batch, num_features] → Embedding [batch, num_features, embed_dim] → Column Attention (features attend to each other) → Row Attention (samples attend to each other) → Repeat for N layers → MLP → Prediction

How It Works

SAINT combines two types of attention for tabular learning:

  1. Self-attention over features (column attention, like FT-Transformer)
  2. Inter-sample attention (row attention, comparing samples within a batch)

This implementation follows the standard neural network architecture pattern with:

  • Proper inheritance from NeuralNetworkBase
  • Layer-based architecture using ILayer components
  • Engine-based tensor operations for CPU/GPU acceleration
  • Full autodiff and JIT compilation support

Reference: “SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training” (2021)

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var options = new SAINTOptions<double> { NumFeatures = 20, EmbeddingDimension = 32, NumHeads = 8, NumLayers = 6 };
var input = Tensor<float>.CreateRandom(new[] { 1, 20 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new SAINTNetwork<float>(architecture))
    .Build(trainX, trainY);
var output = result.Predict(input);

Constructors

Constructor Summary
SAINTNetwork() Initializes a new SAINT network with the specified architecture.
SAINTNetwork(NeuralNetworkArchitecture<T> architecture, SAINTOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction)

Properties

Property Summary
SAINTOptions<T> Options { get; } Gets the SAINT-specific options.
bool UseIntersampleAttention { get; } Gets whether inter-sample (row) attention is enabled.
int EmbeddingDimension { get; } Gets the embedding dimension.
int NumHeads { get; } Gets the number of attention heads.
int NumLayers { get; } Gets the number of transformer layers.

Methods

Method Summary
Dictionary<string, T> GetFeatureImportance() Gets the learned feature importance from the attention layers.
ModelMetadata<T> GetModelMetadata()