Transformer<T>
Models & Types · AiDotNet.NeuralNetworks
Represents a Transformer neural network architecture, which is particularly effective for sequence-based tasks like natural language processing.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → Transformer<T>
Implements: IAuxiliaryLossLayer<T>, ILanguageModel<T>
For Beginners
A Transformer is a modern type of neural network that excels at understanding sequences of data, like sentences or time series.
Think of it like reading a book:
- When you read a sentence, some words are more important than others for understanding the meaning
- A Transformer can “pay attention” to different words based on their importance
- It can look at the entire context at once, rather than reading one word at a time
For example, in the sentence “The animal didn’t cross the street because it was too wide”, the Transformer can figure out that “it” refers to “the street” by paying attention to the relationship between these words.
Transformers are behind many recent AI advances, including large language models like GPT and BERT.
How It Works
The Transformer architecture is a type of neural network design that uses self-attention mechanisms instead of recurrence or convolution. This approach allows the model to weigh the importance of different parts of the input sequence when producing each part of the output sequence.
The key components of a Transformer include:
- Multi-head attention layers: Allow the model to focus on different parts of the input
- Feed-forward networks: Process the attended information
- Layer normalization: Stabilize the network during training
- Residual connections: Help information flow through the network
Example
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var input = Tensor<float>.CreateRandom(new[] { 1, 128, 512 });
var architecture = new TransformerArchitecture<float>(
InputType.OneDimensional, NeuralNetworkTaskType.SequenceClassification,
numEncoderLayers: 0, numDecoderLayers: 2, numHeads: 2,
modelDimension: 32, feedForwardDimension: 64,
NetworkComplexity.Simple, inputSize: 64, outputSize: 64);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new Transformer<float>(architecture))
.Build(trainX, trainY);
var output = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
Transformer(TransformerArchitecture<T> architecture, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, TransformerOptions options) |
Creates a new Transformer neural network with the specified architecture. |
Properties
| Property | Summary |
|---|---|
T AuxiliaryLossWeight { get; set; } |
Gets or sets the weight for the attention regularization auxiliary loss. |
Tensor<T> AttentionMask { get; set; } |
Gets or sets the attention mask used in the Transformer. |
bool UseAuxiliaryLoss { get; set; } |
Gets or sets whether auxiliary loss (attention regularization) should be used during training. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetAuxiliaryLossDiagnostics() |
Gets diagnostic information about the attention regularization auxiliary loss. |
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about this component’s state and behavior. |
ModelMetadata<T> GetModelMetadata() |
Retrieves metadata about the Transformer model. |
T ComputeAuxiliaryLoss() |
Computes the auxiliary loss for attention regularization across all attention layers. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
Training forward. |
Tensor<T> ForwardLogits(Tensor<T> tokenIds) |
|
void SetAttentionMask(Tensor<T> mask) |
Sets the attention mask for the Transformer. |