DocFormerEncoderLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

DocFormer’s multi-modal self-attention encoder (reference DocFormerEncoder + MultiModalAttentionLayer).

Inheritance: LayerBase<T> → DocFormerEncoderLayer<T>

Implements: IShapeContract

How It Works

Each layer sees text features t, visual features v, and the two streams’ spatial features ts, vs, all [L, hidden]. The stream scores are q k / sqrt(dh) + q . a_lr + k_r . a_lr + q_s k_s / sqrt(dh), where a is the stream’s clipped relative-position table and the spatial projections q_s, k_s are shared by the two streams. The layer output is out(softmax(S_t) V_t + softmax(S_v) V_v).

A pre-LN residual wraps each layer. It adds the skip t + v + ts + vs and then the GELU FFN, and only t is carried to the next layer: v, ts and vs are re-injected unchanged from layer 0, which is the paper’s multi-modal feature re-use.

Two choices follow the paper rather than the unofficial port. The attention is scaled by sqrt(head_dim), where the port uses sqrt(hidden). The image stream has its own relative table, where the port reuses the text table.

The single-input forward takes t with the other streams zero.

Constructors

Constructor Summary
DocFormerEncoderLayer(int hidden, int numHeads, int ffnDim, int maxRelative, int numLayers) Creates the encoder.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).