ExternalMemory<T>

Models & Types · AiDotNet.MetaLearning.Algorithms

The external memory of a MANN with least-recently-used access (LRUA), Santoro et al. 2016, section 3.2.

How It Works

The memory M holds one key row per slot; alongside it, a value row per slot carries what was written with that key (the label of the example in MANN’s classification setting), written with the same weights. Per time step, with key k_t:

Ties in usage go to the lower slot index, so the least-used slot and the n least used are deterministic. Every step is made of engine ops, so on a gradient tape the reads and writes are differentiable in the keys, the values and the gate; the least-used selection is piecewise constant, as in the paper.

Constructors

Constructor Summary
ExternalMemory(int size, int keySize, int valueSize, INumericOperations<T> numOps) Initializes a new instance of the ExternalMemory with the default LRUA settings: usage decay 0.95, write gate 0.5 and one read.
ExternalMemory(int size, int keySize, int valueSize, INumericOperations<T> numOps, double usageDecay, double writeGate, int numReads) Initializes a new instance of the ExternalMemory.

Properties

Property Summary
double UsageDecay { get; } Gets the usage decay gamma.
double WriteGate { get; set; } Gets the write gate sigmoid(alpha) used by WriteAndRead; MANN keeps it at its learned value.
int KeySize { get; } Gets the key dimension.
int NumReads { get; } Gets the number of reads n, the number of slots the least-used weights mark.
int Size { get; } Gets the memory size.
int ValueSize { get; } Gets the value dimension.

Methods

Method Summary
ExternalMemory<T> Clone() Clones the external memory.
List<int> FindRarelyUsedSlots(double threshold) Finds the slots whose usage is below threshold.
T ComputeUsagePenalty() Computes memory usage penalty for regularization: the mean usage weight.
Vector<T> ContentWeights(Vector<T> key) The read weights softmax_i(cosine(key, M(i))) for a key, without changing the memory.
Vector<T> GetKey(int index) Gets memory key at specified index.
Vector<T> GetUsage() A copy of the usage weights w^u.
Vector<T> LeastUsedWeights() The least-used weights w^lu: 1 for the NumReads least-used slots, else 0.
Vector<T> Read(Vector<T> attentionWeights) Reads from memory using attention weights: sum_i w(i) V(i) over the first slots the weights cover.
Vector<T> ReadAndUpdate(Vector<T> key) One LRUA read without a write: content weights for key, which update the usage and become the next write’s read weights.
Vector<T> WriteAndRead(Vector<T> key, Vector<T> value) One LRUA time step: writes key and value through the write weights, then reads with the same key.
int FindLeastRecentlyUsedSlot() Finds the least-used memory slot (the lowest index among equal usages).
void ClearRecent(double retentionRatio) Keeps the retentionRatio most-used slots and clears the rest (ties keep the lower index).
void ClearSlot(int location) Clears a specific memory slot.
void Write(int location, Vector<T> key, Vector<T> value) Writes a key and value straight into one slot, as a write whose weight is 1 at location: the slot’s leading entries are replaced and its usage grows by 1.