UdopTransformerLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
UDOP’s unified vision-text-layout transformer (Tang et al. 2023; HF UdopForConditionalGeneration): a T5
encoder-decoder whose encoder sees text, image patches and 2-D layout together.
Inheritance: LayerBase<T> → UdopTransformerLayer<T>
Implements: IShapeContract
How It Works
- Layout-induced fusion. Each token’s embedding gets the embedding of the 16x16 image patch that
contains its box centre. Tokens whose box mean is exactly 0 or 1, such as task prompts, are skipped. Patches that no token claimed are appended after the text, each with its grid-cell box.
- 2-D cell embedding. Left/right corners index an x table and top/bottom corners index a y table.
The four lookups are summed onto the encoder input.
- Encoder position bias. Three bucketed biases are summed: T5 1-D over sequence order, plus horizontal
and vertical biases over box-centre distances scaled by 100. The bias is computed once and shared by every block.
- Decoder. A T5 causal 1-D bias from block 0, cross-attention without bias, and a head tied to the
shared embedding with the d^-1/2 rescale.
All blocks are pre-RMSNorm residual blocks with bias-free projections and a ReLU feed-forward.
Constructors
| Constructor | Summary |
|---|---|
UdopTransformerLayer(int vocabSize, int dim, int numHeads, int keyValueDim, int ffDim, int numEncoderLayers, int numDecoderLayers, int imageSize, int patchSize, int numBuckets, int maxDistance, int maxDistance2D, int max2DPositions, double epsilon) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<LayerPort> InputPorts { get; } |
Generated from [TensorPort] declarations. |
IReadOnlyList<LayerPort> OutputPorts { get; } |
Generated from [TensorPort] declarations. |
bool SupportsTraining { get; } |
|
int ImageSize { get; } |
Side of the square page image the patch embedding expects. |
int VocabSize { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
LayerInputDomain GetInputDomain(int[] inputShape) |
Generated from the primary [TensorPort] declaration. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |
Fields
| Field | Summary |
|---|---|
int CoordinateGrid |
Boxes arrive on the 0-1000 grid the UDOP processor emits; the model works on 0-1. |