RobertaEncoderLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

The RoBERTa text encoder (Liu et al. 2019) up to its pooled output, as Hugging Face RobertaModel computes pooler_output: token ids in, one vector per sequence out.

Inheritance: LayerBase<T> → RobertaEncoderLayer<T>

Implements: IShapeContract

For Beginners

This reads a sentence as numbers (token ids), lets every word look at every other word over several rounds, and then summarizes the whole sentence into a single vector taken from the special start-of-sentence token.

How It Works

The computation, layer by layer:

token-type, LayerNorm, dropout. Positions are counted over the non-padding tokens starting at pad + 1.

  • N post-LN encoder layers: biased query/key/value projections, scaled dot-product attention over

heads heads with padding keys masked out, attention-probability dropout, an output projection, dropout, then LayerNorm(x + attention); an intermediate projection with the exact (erf) GELU, an output projection, dropout, then LayerNorm(h + ffn).

  • Pooler: the first (<s>) token’s hidden state through a dense layer with tanh.

The attention mask is derived from the ids: a position is padding exactly when its id equals the padding token id, which is what the RoBERTa tokenizer’s attention_mask encodes for a padded batch (the padding id never occurs inside text).

Constructors

Constructor Summary
RobertaEncoderLayer(int vocabularySize, int hiddenSize, int numLayers, int numHeads, int intermediateSize, int maxPositions, int paddingTokenId, double dropoutRate, double layerNormEpsilon) Creates the encoder.

Properties

Property Summary
BertEmbeddingLayer<T> Embeddings { get; } The embedding block (word, position, token type, LayerNorm).
DenseLayer<T> Pooler { get; } The pooler’s dense layer (tanh), applied to the first token.
IReadOnlyList<LayerPort> InputPorts { get; } Generated from [TensorPort] declarations.
IReadOnlyList<LayerPort> OutputPorts { get; } Generated from [TensorPort] declarations.
bool SupportsTraining { get; }
int HiddenSize { get; } Model width.
int NumLayers { get; } Number of encoder layers.
int PaddingTokenId { get; } The padding token id.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
LayerInputDomain GetInputDomain(int[] inputShape) Generated from the primary [TensorPort] declaration.
void ResetState()