AttentionNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
Represents a neural network that utilizes attention mechanisms for sequence processing.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → SequenceModelLayoutBase<T> → AttentionNetwork<T>
Implements: IAuxiliaryLossLayer<T>
For Beginners
This network mimics how humans pay attention to different parts of information.
Think of it like reading a complex paragraph:
- When you try to understand a sentence, you don’t focus equally on all words
- You focus more on the important words that carry meaning
- You also connect related words even if they’re far apart
For example, in the sentence “The cat, which had a white spot on its tail, chased the mouse”:
- An attention network would connect “cat” with “chased” even though they’re separated
- It would assign different importance to different words based on context
- This helps it understand the overall meaning better than networks that process words in isolation
This ability to selectively focus and connect distant information makes attention networks powerful for language tasks, time series prediction, and many other sequence-based problems.
How It Works
An attention network is a specialized neural network architecture designed for sequence processing tasks. It uses attention mechanisms to dynamically focus on different parts of the input sequence when generating outputs. This allows the network to capture long-range dependencies and relationships between elements in the sequence, making it particularly effective for tasks like natural language processing, time series analysis, and other sequence-to-sequence problems.
Example
var architecture = new NeuralNetworkArchitecture<float>(
InputType.TwoDimensional, NeuralNetworkTaskType.Regression,
inputHeight: 10, inputWidth: 64, outputSize: 4);
var options = new AttentionNetworkOptions { HeadCount = 8, EncoderBlockCount = 3 };
var input = Tensor<float>.CreateRandom(new[] { 10, 64 }); // (sequenceLength, embeddingSize)
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new AttentionNetwork<float>(architecture, sequenceLength: 10, embeddingSize: 64, options: options))
.Build(trainX, trainY);
var output = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
AttentionNetwork() |
Initializes a new instance with default architecture settings. |
AttentionNetwork(NeuralNetworkArchitecture<T> architecture, int sequenceLength, int embeddingSize, ILossFunction<T> lossFunction, AttentionNetworkOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Initializes a new instance of the AttentionNetwork class. |
Properties
| Property | Summary |
|---|---|
T AuxiliaryLossWeight { get; set; } |
Gets or sets the weight for attention entropy regularization. |
bool UseAuxiliaryLoss { get; set; } |
Gets or sets whether to use auxiliary loss (attention entropy regularization) during training. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetAuxiliaryLossDiagnostics() |
Gets diagnostic information about the attention entropy regularization. |
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about this component’s state and behavior. |
ModelMetadata<T> GetModelMetadata() |
Gets metadata about the Attention Network model. |
T ComputeAuxiliaryLoss() |
Computes the auxiliary loss for the AttentionNetwork, which aggregates attention entropy losses from all attention layers. |