ClozeAttentionLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Bidirectional Cloze Network (BCN) attention per Fang et al., CVPR 2021, “Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition” (arXiv:2103.06495).

Inheritance: LayerBase<T> → ClozeAttentionLayer<T>

Implements: IShapeContract

How It Works

This is ABINet’s Bidirectional principle. Each position attends to every OTHER position — both left and right — but is forbidden from attending to itself:

Blocking the diagonal is what makes it a cloze: the representation at position i is built purely from its surrounding context, so predicting the character there cannot trivially copy the character itself. That is precisely the information leak an unmasked bidirectional attention would introduce, and why this cannot be expressed with the causal (triangular) mask an ordinary attention layer offers — a causal mask would also remove all right-hand context, collapsing the model back to unidirectional.

Gradient tracking: the mask is a constant tensor added to the scores, and every other step is an IEngine op, so the tape records the whole block.

Constructors

Constructor Summary
ClozeAttentionLayer(int modelDim) Initializes a new bidirectional cloze attention block.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ResetState()