RobertaEncoderLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
The RoBERTa text encoder (Liu et al. 2019) up to its pooled output, as Hugging Face RobertaModel computes
pooler_output: token ids in, one vector per sequence out.
Inheritance: LayerBase<T> → RobertaEncoderLayer<T>
Implements: IShapeContract
For Beginners
This reads a sentence as numbers (token ids), lets every word look at every other word over several rounds, and then summarizes the whole sentence into a single vector taken from the special start-of-sentence token.
How It Works
The computation, layer by layer:
- Embeddings (
BertEmbeddingLayerin RoBERTa mode): word + learned position +
token-type, LayerNorm, dropout. Positions are counted over the non-padding tokens starting at
pad + 1.
- N post-LN encoder layers: biased query/key/value projections, scaled dot-product attention over
heads heads with padding keys masked out, attention-probability dropout, an output projection, dropout,
then LayerNorm(x + attention); an intermediate projection with the exact (erf) GELU, an output
projection, dropout, then LayerNorm(h + ffn).
- Pooler: the first (
<s>) token’s hidden state through a dense layer with tanh.
The attention mask is derived from the ids: a position is padding exactly when its id equals the padding
token id, which is what the RoBERTa tokenizer’s attention_mask encodes for a padded batch (the padding
id never occurs inside text).
Constructors
| Constructor | Summary |
|---|---|
RobertaEncoderLayer(int vocabularySize, int hiddenSize, int numLayers, int numHeads, int intermediateSize, int maxPositions, int paddingTokenId, double dropoutRate, double layerNormEpsilon) |
Creates the encoder. |
Properties
| Property | Summary |
|---|---|
BertEmbeddingLayer<T> Embeddings { get; } |
The embedding block (word, position, token type, LayerNorm). |
DenseLayer<T> Pooler { get; } |
The pooler’s dense layer (tanh), applied to the first token. |
IReadOnlyList<LayerPort> InputPorts { get; } |
Generated from [TensorPort] declarations. |
IReadOnlyList<LayerPort> OutputPorts { get; } |
Generated from [TensorPort] declarations. |
bool SupportsTraining { get; } |
|
int HiddenSize { get; } |
Model width. |
int NumLayers { get; } |
Number of encoder layers. |
int PaddingTokenId { get; } |
The padding token id. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Generated from the primary [TensorPort] declaration. |
void ResetState() |