SAINTOptions<T>

Options & Configuration · AiDotNet.Models.Options

Configuration options for SAINT (Self-Attention and Intersample Attention Transformer).

Inheritance: ModelOptions → NeuralNetworkOptions → ModelHyperparameterOptions → FinancialNeuralNetworkOptions → RiskModelOptions<T> → SAINTOptions<T>

For Beginners

SAINT is powerful because it learns two things:

  • Column attention: Which features are related to each other?
  • Row attention: Which training samples are similar?

This dual attention makes SAINT especially good when similar samples share patterns that are useful for prediction.

Example:

How It Works

SAINT combines two types of attention:

  1. Self-attention over features (column attention, like FT-Transformer)
  2. Inter-sample attention (row attention, comparing samples within a batch)

Reference: “SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training” (2021)

Constructors

Constructor Summary
SAINTOptions()

Properties

Property Summary
IActivationFunction<T> HiddenActivation { get; set; } Gets or sets the hidden layer activation function.
IVectorActivationFunction<T> HiddenVectorActivation { get; set; } Gets or sets the hidden layer vector activation function (alternative to scalar activation).
bool UseIntersampleAttention { get; set; } Gets or sets whether to use inter-sample (row) attention.
bool UseLayerNorm { get; set; } Gets or sets whether to use layer normalization.
bool UsePreNorm { get; set; } Gets or sets whether to use pre-norm (norm before attention) or post-norm.
double DropoutRate { get; set; } Dropout rate.
double EmbeddingInitScale { get; set; } Gets or sets the initialization scale for embeddings.
int BatchSize { get; set; } Batch size (sequence length for inter-sample attention).
int EmbeddingDimension { get; set; } Gets or sets the embedding dimension for features.
int FeedForwardDimension { get; } Gets the feed-forward network dimension.
int FeedForwardMultiplier { get; set; } Gets or sets the feed-forward dimension multiplier.
int HiddenDimension { get; set; } Hidden dimension size for feed-forward networks.
int NumHeads { get; set; } Number of attention heads.
int NumLayers { get; set; } Number of transformer layers.
int[] CategoricalCardinalities { get; set; } Gets or sets the cardinalities of categorical features.
int[] MLPHiddenDimensions { get; set; } Gets or sets the hidden dimensions for the MLP head.

Methods

Method Summary
void Validate() Validates that the options are consistent (e.g., NumHeads evenly divides EmbeddingDimension).