UdopTransformerLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

UDOP’s unified vision-text-layout transformer (Tang et al. 2023; HF UdopForConditionalGeneration): a T5 encoder-decoder whose encoder sees text, image patches and 2-D layout together.

Inheritance: LayerBase<T> → UdopTransformerLayer<T>

Implements: IShapeContract

How It Works

  • Layout-induced fusion. Each token’s embedding gets the embedding of the 16x16 image patch that

contains its box centre. Tokens whose box mean is exactly 0 or 1, such as task prompts, are skipped. Patches that no token claimed are appended after the text, each with its grid-cell box.

  • 2-D cell embedding. Left/right corners index an x table and top/bottom corners index a y table.

The four lookups are summed onto the encoder input.

  • Encoder position bias. Three bucketed biases are summed: T5 1-D over sequence order, plus horizontal

and vertical biases over box-centre distances scaled by 100. The bias is computed once and shared by every block.

  • Decoder. A T5 causal 1-D bias from block 0, cross-attention without bias, and a head tied to the

shared embedding with the d^-1/2 rescale.

All blocks are pre-RMSNorm residual blocks with bias-free projections and a ReLU feed-forward.

Constructors

Constructor Summary
UdopTransformerLayer(int vocabSize, int dim, int numHeads, int keyValueDim, int ffDim, int numEncoderLayers, int numDecoderLayers, int imageSize, int patchSize, int numBuckets, int maxDistance, int maxDistance2D, int max2DPositions, double epsilon)

Properties

Property Summary
IReadOnlyList<LayerPort> InputPorts { get; } Generated from [TensorPort] declarations.
IReadOnlyList<LayerPort> OutputPorts { get; } Generated from [TensorPort] declarations.
bool SupportsTraining { get; }
int ImageSize { get; } Side of the square page image the patch embedding expects.
int VocabSize { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
LayerInputDomain GetInputDomain(int[] inputShape) Generated from the primary [TensorPort] declaration.
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).

Fields

Field Summary
int CoordinateGrid Boxes arrive on the 0-1000 grid the UDOP processor emits; the model works on 0-1.