MemoryWriteLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Represents a layer that writes to a memory tensor using an attention mechanism.
Inheritance: LayerBase<T> → MemoryWriteLayer<T>
Implements: IAuxiliaryLossLayer<T>, IShapeContract
For Beginners
This layer helps a neural network store information in memory.
Think of it like deciding what to write in a notebook:
- You have some new information (your current input)
- You have a notebook with existing notes (your memory)
- The layer decides which pages of the notebook are relevant to your new information
- It then writes the new information on those pages, focusing more on the most relevant ones
For example, if your input represents new information about “France has a beautiful capital city”, the layer would focus on memory locations related to France and update them with this new information.
This is similar to how we humans selectively update our memories with new information, rather than storing everything in completely new locations.
How It Works
The MemoryWriteLayer implements a form of attention-based memory writing. It computes attention scores between the input and memory tensors, using these scores to determine where to write new information. This approach allows the layer to selectively update memory based on the current input. The layer uses a query-key-value attention mechanism where queries and keys determine where to write, and values determine what to write.
Constructors
| Constructor | Summary |
|---|---|
MemoryWriteLayer(int memoryDimension, IActivationFunction<T> activationFunction) |
Initializes a new instance of the MemoryWriteLayer class with the specified dimensions and a vector activation function. |
MemoryWriteLayer(int memoryDimension, IVectorActivationFunction<T> activationFunction) |
Initializes a new instance of the MemoryWriteLayer class with the specified dimensions and a vector activation function. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<LayerPort> InputPorts { get; } |
|
T AuxiliaryLossWeight { get; set; } |
Gets or sets the weight for the auxiliary loss contribution. |
bool SupportsTraining { get; } |
|
bool UseAuxiliaryLoss { get; set; } |
Gets or sets a value indicating whether auxiliary loss is enabled for this layer. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetAuxiliaryLossDiagnostics() |
Gets diagnostic information about the auxiliary loss computation. |
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about this component’s state and behavior. |
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
T ComputeAuxiliaryLoss() |
Computes the auxiliary loss for this layer based on attention sparsity regularization. |
Tensor<T> Forward(Tensor<T> input, Tensor<T> memory) |
Performs the forward pass of the memory write layer with input and memory tensors. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates the parameters of the memory write layer using the calculated gradients. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |