SwinTransformerBlockLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Swin Transformer block layer with windowed multi-head self-attention.
Inheritance: LayerBase<T> → SwinTransformerBlockLayer<T>
Implements: IShapeContract
For Beginners
Unlike standard transformers that compute attention across all tokens (which is expensive for images), Swin Transformer divides the image into windows and computes attention only within each window. To allow information flow between windows, alternate layers use “shifted” windows that overlap the original window boundaries.
How It Works
This layer implements the core Swin Transformer block with:
- Window-based multi-head self-attention (W-MSA or SW-MSA)
- Shifted window partitioning for cross-window connections
- Two-layer MLP with GELU activation
- Pre-norm architecture with residual connections
- Learnable relative position bias
Reference: Liu et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows”, ICCV 2021
Constructors
| Constructor | Summary |
|---|---|
SwinTransformerBlockLayer(int dim, int numHeads, int windowSize, int shiftSize, int mlpRatio, double dropPathRate, bool exactGelu) |
Creates a new Swin Transformer block layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
bool UsesExactGelu { get; } |
Whether the MLP uses the exact (erf) GELU rather than the tanh approximation. |
bool UsesShiftedWindows { get; } |
Gets whether this block uses shifted windows. |
int WindowSize { get; } |
The attention window side. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |