MemoryReadLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Represents a layer that reads from a memory tensor using an attention mechanism.
Inheritance: LayerBase<T> → MemoryReadLayer<T>
Implements: IAuxiliaryLossLayer<T>, IShapeContract
For Beginners
This layer helps a neural network retrieve information from memory.
Think of it like searching for relevant information in a book:
- You have a query (your current input)
- You have a memory (like pages of a book)
- The layer finds which parts of the memory are most relevant to your query
- It then combines those relevant parts to produce an output
For example, if your input represents a question like “What’s the capital of France?”, the layer would look through memory to find information about France, give more attention to content about its capital, and then combine this information to produce the answer “Paris”.
This is similar to how modern language models can retrieve and use stored information when answering questions.
How It Works
The MemoryReadLayer implements a form of attention-based memory access. It computes attention scores between the input and memory tensors, using these scores to create a weighted sum of memory values. This approach allows the layer to selectively retrieve information from memory based on the current input. The layer consists of key weights (for attention computation), value weights (for transforming memory values), and output weights (for final processing).
Constructors
| Constructor | Summary |
|---|---|
MemoryReadLayer(int memoryDimension, int outputDimension, IActivationFunction<T> activationFunction) |
Initializes a new instance of the MemoryReadLayer class with the specified dimensions and a vector activation function. |
MemoryReadLayer(int memoryDimension, int outputDimension, IVectorActivationFunction<T> activationFunction) |
Initializes a new instance of the MemoryReadLayer class with the specified dimensions and a vector activation function. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<LayerPort> InputPorts { get; } |
Declares named input ports for this multi-input layer. |
T AuxiliaryLossWeight { get; set; } |
Gets or sets the weight for the auxiliary loss contribution. |
bool SupportsTraining { get; } |
|
bool UseAuxiliaryLoss { get; set; } |
Gets or sets a value indicating whether auxiliary loss is enabled for this layer. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetAuxiliaryLossDiagnostics() |
Gets diagnostic information about the auxiliary loss computation. |
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about this component’s state and behavior. |
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
T ComputeAuxiliaryLoss() |
Computes the auxiliary loss for this layer based on attention sparsity regularization. |
Tensor<T> Forward(Tensor<T> input, Tensor<T> memory) |
Performs the forward pass of the memory read layer with input and memory tensors. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates the parameters of the memory read layer using the calculated gradients. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |