DocFormerSpatialEmbeddingLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

DocFormer’s spatial embedding of one stream (Appalaraju et al., ICCV 2021; reference DocFormerEmbeddings).

Inheritance: LayerBase<T> → DocFormerSpatialEmbeddingLayer<T>

Implements: IShapeContract

For Beginners

This turns each word’s box, and where it sits relative to the word before it, into a vector the attention can compare.

How It Works

Per axis, a token is described by eight lookups:

  • Its two edges (tables of width coordinateSize).
  • Its extent.
  • The distances of its four corners and its centroid to the previous token’s (tables of width

coordinateSize; distances are clamped to +-maxPosition2D).

The eight are concatenated. The x and y concatenations and a sinusoidal position are then summed, giving 8 x coordinateSize = hidden. The visual and text streams each own one of these layers, as in the reference.

Input: boxes [S, 4] as integer (x0, y0, x1, y1) on the 0 .. maxPosition2D - 1 grid.

Constructors

Constructor Summary
DocFormerSpatialEmbeddingLayer(int hidden, int maxPosition2D) Creates the embedding.

Properties

Property Summary
IReadOnlyList<LayerPort> InputPorts { get; } Generated from [TensorPort] declarations.
IReadOnlyList<LayerPort> OutputPorts { get; } Generated from [TensorPort] declarations.
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
LayerInputDomain GetInputDomain(int[] inputShape) Generated from the primary [TensorPort] declaration.
void ResetState()