ExperienceReplayBuffer<T, TInput, TOutput>

Models & Types · AiDotNet.ContinualLearning.Memory

A memory buffer for storing examples from previous tasks for experience replay.

Inheritance: ModelBase<T, TInput, TOutput> → ExperienceReplayBuffer<T, TInput, TOutput>

For Beginners

Experience replay stores a small number of examples from previous tasks and intermixes them with new task data during training. This helps prevent catastrophic forgetting by reminding the model of what it learned before.

How It Works

Why It Works: When a neural network learns a new task, it adjusts its weights to minimize error on that task. Without replay, these adjustments can increase error on previous tasks. By mixing old examples with new training data, the network must maintain performance on both old and new tasks.

Memory Management: Since storing all examples is impractical, we use smart sampling strategies to select the most representative examples. Different strategies work better for different scenarios:

  • Reservoir: Fair representation, good for general use
  • ClassBalanced: Equal class representation, best for imbalanced data
  • Herding: Class-mean exemplars, from iCaRL paper

References:

  • Chaudhry et al. “Continual Learning with Tiny Episodic Memories” (2019)
  • Rebuffi et al. “iCaRL: Incremental Classifier and Representation Learning” (2017)
  • Rolnick et al. “Experience Replay for Continual Learning” (2019)

Constructors

Constructor Summary
ExperienceReplayBuffer(int maxSize, MemorySamplingStrategy addStrategy, ReplaySamplingStrategy replayStrategy, int? seed) Initializes a new experience replay buffer.

Properties

Property Summary
ILossFunction<T> DefaultLossFunction { get; }
MemorySamplingStrategy AddStrategy { get; } Gets the sampling strategy used when adding examples.
ReplaySamplingStrategy ReplayStrategy { get; } Gets the sampling strategy used during replay.
bool IsFull { get; } Gets whether the buffer is at capacity.
int Count { get; } Gets the current number of stored examples.
int MaxSize { get; } Gets the maximum capacity of the buffer.
int TaskCount { get; } Gets the number of distinct tasks represented in the buffer.
int TotalReplaySamples { get; } Gets the total number of samples returned via replay.
int TotalSamplesProcessed { get; } Gets the total number of samples processed (added) since creation.
long EstimatedMemoryBytes { get; } Gets the estimated memory usage in bytes.

Methods

Method Summary
BufferStatistics GetStatistics() Gets statistics about the buffer.
IFullModel<T, TInput, TOutput> WithParameters(Vector<T> parameters)
IReadOnlyDictionary<int, int> GetTaskCounts() Gets the count of examples per task.
IReadOnlyList<DataPoint<T, TInput, TOutput>> GetAll() Gets all stored examples.
IReadOnlyList<DataPoint<T, TInput, TOutput>> GetTaskExamples(int taskId) Gets all examples for a specific task.
List<DataPoint<T, TInput, TOutput>> SampleBatch(int batchSize) Samples a batch of examples from the buffer using the configured replay strategy.
List<DataPoint<T, TInput, TOutput>> SampleFromTask(int taskId, int count) Samples examples from a specific task.
TOutput Predict(TInput input)
void AddTaskExamples(IDataset<T, TInput, TOutput> taskData, int taskId, int? samplesPerTask) Adds examples from a task to the buffer using the configured sampling strategy.
void Clear() Clears all stored examples.
void RemoveTask(int taskId) Removes all examples from a specific task.
void Train(TInput input, TOutput expectedOutput)
void UpdatePriority(int index, double priority) Updates the priority of a sample (for priority-based replay).