TransformerArchitecture<T>
Models & Types · AiDotNet.NeuralNetworks
Defines the architecture configuration for a Transformer neural network.
Inheritance: NeuralNetworkArchitecture<T> → TransformerArchitecture<T>
For Beginners
Think of this class as a blueprint for building a Transformer.
Just like building a house requires decisions about how many rooms, how big each room should be, and what materials to use, building a Transformer requires decisions about:
- How many layers of processing to include
- How much information to process at once
- How to connect different parts of the network
This class stores all those decisions in one place, making it easier to create Transformer networks with different capabilities for different tasks.
How It Works
The TransformerArchitecture class encapsulates all the hyperparameters and configuration options needed to define a Transformer neural network. It includes settings for the encoder and decoder stacks, attention mechanisms, model dimensions, and other key aspects that determine the network’s structure and behavior.
Transformers are particularly effective for sequence-based tasks like natural language processing, translation, text summarization, and other tasks that benefit from understanding the relationships between elements in a sequence.
Constructors
| Constructor | Summary |
|---|---|
TransformerArchitecture(InputType inputType, NeuralNetworkTaskType taskType, int numEncoderLayers, int numDecoderLayers, int numHeads, int modelDimension, int feedForwardDimension, NetworkComplexity complexity, int inputSize, int outputSize, double dropoutRate, int maxSequenceLength, int vocabularySize, bool usePositionalEncoding, double temperature, SequencePoolingMode? sequencePooling, List<ILayer<T>> layers) |
Initializes a new instance of the TransformerArchitecture class with the specified parameters. |
TransformerArchitecture(InputType inputType, NeuralNetworkTaskType taskType, int numEncoderLayers, int numDecoderLayers, int numHeads, int modelDimension, int feedForwardDimension, NetworkComplexity complexity, int inputSize, int outputSize, double dropoutRate, int maxSequenceLength, int vocabularySize, bool usePositionalEncoding, double temperature, SequencePoolingMode? sequencePooling, List<ILayer<T>> layers, int warmupSteps, int? randomSeed) |
Properties
| Property | Summary |
|---|---|
SequencePoolingMode SequencePooling { get; } |
Strategy for collapsing the encoder’s [batch, seq, dim] hidden states into a single [batch, dim] vector before the classification head, when the task is single-label per sequence. |
bool UsePositionalEncoding { get; } |
Gets a value indicating whether positional encoding is used in the Transformer. |
double DropoutRate { get; } |
Gets the dropout rate used for regularization in the Transformer. |
double Temperature { get; } |
Gets the temperature parameter used for controlling randomness in text generation. |
int FeedForwardDimension { get; } |
Gets the dimension of the feed-forward networks within the Transformer layers. |
int MaxSequenceLength { get; } |
Gets the maximum length of input sequences that the Transformer can process. |
int ModelDimension { get; } |
Gets the dimension of the model’s internal representations. |
int NumDecoderLayers { get; } |
Gets the number of decoder layers in the Transformer. |
int NumEncoderLayers { get; } |
Gets the number of encoder layers in the Transformer. |
int NumHeads { get; } |
Gets the number of attention heads in each multi-head attention layer. |
int VocabularySize { get; } |
Gets the size of the vocabulary for text-based tasks. |
int WarmupSteps { get; } |
Gets the number of warmup steps for the default Noam (Vaswani 2017) learning-rate schedule attached to the Transformer’s default Adam optimizer. |