SwinTransformerBlockLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Swin Transformer block layer with windowed multi-head self-attention.

Inheritance: LayerBase<T> → SwinTransformerBlockLayer<T>

Implements: IShapeContract

For Beginners

Unlike standard transformers that compute attention across all tokens (which is expensive for images), Swin Transformer divides the image into windows and computes attention only within each window. To allow information flow between windows, alternate layers use “shifted” windows that overlap the original window boundaries.

How It Works

This layer implements the core Swin Transformer block with:

  • Window-based multi-head self-attention (W-MSA or SW-MSA)
  • Shifted window partitioning for cross-window connections
  • Two-layer MLP with GELU activation
  • Pre-norm architecture with residual connections
  • Learnable relative position bias

Reference: Liu et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows”, ICCV 2021

Constructors

Constructor Summary
SwinTransformerBlockLayer(int dim, int numHeads, int windowSize, int shiftSize, int mlpRatio, double dropPathRate, bool exactGelu) Creates a new Swin Transformer block layer.

Properties

Property Summary
bool SupportsTraining { get; }
bool UsesExactGelu { get; } Whether the MLP uses the exact (erf) GELU rather than the tanh approximation.
bool UsesShiftedWindows { get; } Gets whether this block uses shifted windows.
int WindowSize { get; } The attention window side.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).