STCConnectorLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Implements the Spatial-Temporal Convolution (STC) connector from VideoLLaMA 2 (Cheng et al. 2024, arXiv:2406.07476).

Inheritance: LayerBase<T> → STCConnectorLayer<T>

Implements: IShapeContract

How It Works

The paper’s connector is not a lone 3D convolution. Its published structure is RegStage -> Conv3D downsampler -> RegStage -> MLP. Each RegStage uses depthwise RegNet-style bottleneck blocks with LayerNorm2d and SiLU. The default constructor values reproduce the released connector: stage depth 4, a 2 x 2 x 2 convolution with stride 2 and padding 1, and a two-layer GELU MLP.

Token input may be unbatched [frames, Hp*Wp, visionDim], batched [batch, frames, Hp*Wp, visionDim], or an already-expanded grid [batch, frames, Hp, Wp, visionDim]. The output is the flattened visual-token sequence in the language decoder width.

Constructors

Constructor Summary
STCConnectorLayer(int dim, int patchesHeight, int patchesWidth, int kernelSize, int stride, int padding) Creates a connector that preserves the input/output feature width.
STCConnectorLayer(int visionDim, int decoderDim, int patchesHeight, int patchesWidth, int kernelSize, int stride, int padding, int stageDepth, int mlpDepth) Creates the paper’s complete STC connector.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
void ClearGradients()
void ResetState()
void SetTrainingMode(bool isTraining)
void UpdateParameters(T learningRate)