ExternalMemory<T>
Models & Types · AiDotNet.MetaLearning.Algorithms
The external memory of a MANN with least-recently-used access (LRUA), Santoro et al. 2016, section 3.2.
How It Works
The memory M holds one key row per slot; alongside it, a value row per slot carries what was written with
that key (the label of the example in MANN’s classification setting), written with the same weights. Per time
step, with key k_t:
Ties in usage go to the lower slot index, so the least-used slot and the n least used are deterministic. Every step is made of engine ops, so on a gradient tape the reads and writes are differentiable in the keys, the values and the gate; the least-used selection is piecewise constant, as in the paper.
Constructors
| Constructor | Summary |
|---|---|
ExternalMemory(int size, int keySize, int valueSize, INumericOperations<T> numOps) |
Initializes a new instance of the ExternalMemory with the default LRUA settings: usage decay 0.95, write gate 0.5 and one read. |
ExternalMemory(int size, int keySize, int valueSize, INumericOperations<T> numOps, double usageDecay, double writeGate, int numReads) |
Initializes a new instance of the ExternalMemory. |
Properties
| Property | Summary |
|---|---|
double UsageDecay { get; } |
Gets the usage decay gamma. |
double WriteGate { get; set; } |
Gets the write gate sigmoid(alpha) used by WriteAndRead; MANN keeps it at its learned value. |
int KeySize { get; } |
Gets the key dimension. |
int NumReads { get; } |
Gets the number of reads n, the number of slots the least-used weights mark. |
int Size { get; } |
Gets the memory size. |
int ValueSize { get; } |
Gets the value dimension. |
Methods
| Method | Summary |
|---|---|
ExternalMemory<T> Clone() |
Clones the external memory. |
List<int> FindRarelyUsedSlots(double threshold) |
Finds the slots whose usage is below threshold. |
T ComputeUsagePenalty() |
Computes memory usage penalty for regularization: the mean usage weight. |
Vector<T> ContentWeights(Vector<T> key) |
The read weights softmax_i(cosine(key, M(i))) for a key, without changing the memory. |
Vector<T> GetKey(int index) |
Gets memory key at specified index. |
Vector<T> GetUsage() |
A copy of the usage weights w^u. |
Vector<T> LeastUsedWeights() |
The least-used weights w^lu: 1 for the NumReads least-used slots, else 0. |
Vector<T> Read(Vector<T> attentionWeights) |
Reads from memory using attention weights: sum_i w(i) V(i) over the first slots the weights cover. |
Vector<T> ReadAndUpdate(Vector<T> key) |
One LRUA read without a write: content weights for key, which update the usage and become the next write’s read weights. |
Vector<T> WriteAndRead(Vector<T> key, Vector<T> value) |
One LRUA time step: writes key and value through the write weights, then reads with the same key. |
int FindLeastRecentlyUsedSlot() |
Finds the least-used memory slot (the lowest index among equal usages). |
void ClearRecent(double retentionRatio) |
Keeps the retentionRatio most-used slots and clears the rest (ties keep the lower index). |
void ClearSlot(int location) |
Clears a specific memory slot. |
void Write(int location, Vector<T> key, Vector<T> value) |
Writes a key and value straight into one slot, as a write whose weight is 1 at location: the slot’s leading entries are replaced and its usage grows by 1. |