ClozeAttentionLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Bidirectional Cloze Network (BCN) attention per Fang et al., CVPR 2021, “Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition” (arXiv:2103.06495).
Inheritance: LayerBase<T> → ClozeAttentionLayer<T>
Implements: IShapeContract
How It Works
This is ABINet’s Bidirectional principle. Each position attends to every OTHER position — both left and right — but is forbidden from attending to itself:
Blocking the diagonal is what makes it a cloze: the representation at position
i is built purely from its surrounding context, so predicting the character there
cannot trivially copy the character itself. That is precisely the information leak an
unmasked bidirectional attention would introduce, and why this cannot be expressed with the
causal (triangular) mask an ordinary attention layer offers — a causal mask would also remove
all right-hand context, collapsing the model back to unidirectional.
Gradient tracking: the mask is a constant tensor added to the scores, and every
other step is an IEngine op, so the tape records the whole block.
Constructors
| Constructor | Summary |
|---|---|
ClozeAttentionLayer(int modelDim) |
Initializes a new bidirectional cloze attention block. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ResetState() |