AttentiveTransformerLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Implements the Attentive Transformer block used in TabNet architecture for feature selection.

Inheritance: LayerBase<T> → AttentiveTransformerLayer<T>

Implements: IShapeContract

For Beginners

The Attentive Transformer is the “feature selector” in TabNet.

At each decision step, it answers the question: “Which features should I focus on next?”

Key concepts:

  • Takes processed features and a “prior scales” tensor as input
  • Prior scales track which features have already been used in previous steps
  • Outputs a sparse attention mask (many values are exactly 0)
  • Features with high attention values are selected for processing
  • Features with 0 attention are completely ignored

This sparse attention provides interpretability - you can see exactly which features the model considers important for each prediction.

How It Works

The Attentive Transformer learns which features to pay attention to at each decision step. It produces sparse attention masks using Sparsemax, ensuring that only the most relevant features are selected for processing.

The attention mechanism uses:

  1. Fully connected layer to compute attention logits
  2. Ghost Batch Normalization for regularization
  3. Prior scaling to discourage feature reuse
  4. Sparsemax to produce sparse probability distribution

Reference: “TabNet: Attentive Interpretable Tabular Learning” (Arik & Pfister, AAAI 2021)

Constructors

Constructor Summary
AttentiveTransformerLayer(int inputDim, int outputDim, double relaxationFactor, int virtualBatchSize, double momentum, double epsilon) Initializes a new instance of the AttentiveTransformer class.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
T ComputeSparsityLoss(Tensor<T> attentionMask) Computes the sparsity regularization loss.
Tensor<T> Forward(Tensor<T> processedFeatures, Tensor<T> priorScales) Performs the forward pass to compute sparse attention mask.
Tensor<T> GetAttentionMask() Gets the last computed attention mask.
Tensor<T> GetBiases()
Tensor<T> GetWeights()
Tensor<T> UpdatePriorScales(Tensor<T> priorScales, Tensor<T> attentionMask) Updates the prior scales based on the current attention mask.
int[] GetInputShape() Gets the input shape.
void ClearGradients()
void ResetState()
void SetTrainingMode(bool isTraining) Sets training mode.
void UpdateParameters(T learningRate)
void UpdateParameters(Vector<T> parameters)