STCConnectorLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Implements the Spatial-Temporal Convolution (STC) connector from VideoLLaMA 2 (Cheng et al. 2024, arXiv:2406.07476).
Inheritance: LayerBase<T> → STCConnectorLayer<T>
Implements: IShapeContract
How It Works
The paper’s connector is not a lone 3D convolution. Its published structure is
RegStage -> Conv3D downsampler -> RegStage -> MLP. Each RegStage uses
depthwise RegNet-style bottleneck blocks with LayerNorm2d and SiLU. The default
constructor values reproduce the released connector: stage depth 4, a
2 x 2 x 2 convolution with stride 2 and padding 1, and a two-layer GELU MLP.
Token input may be unbatched [frames, Hp*Wp, visionDim], batched
[batch, frames, Hp*Wp, visionDim], or an already-expanded grid
[batch, frames, Hp, Wp, visionDim]. The output is the flattened visual-token
sequence in the language decoder width.
Constructors
| Constructor | Summary |
|---|---|
STCConnectorLayer(int dim, int patchesHeight, int patchesWidth, int kernelSize, int stride, int padding) |
Creates a connector that preserves the input/output feature width. |
STCConnectorLayer(int visionDim, int decoderDim, int patchesHeight, int patchesWidth, int kernelSize, int stride, int padding, int stageDepth, int mlpDepth) |
Creates the paper’s complete STC connector. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
void ClearGradients() |
|
void ResetState() |
|
void SetTrainingMode(bool isTraining) |
|
void UpdateParameters(T learningRate) |