TrainingMemoryManager<T>

Models & Types · AiDotNet.Training.Memory

Manages memory optimization during neural network training including gradient checkpointing, activation pooling, and model sharding.

Implements: IDisposable

For Beginners

This manager helps you train larger neural networks by:

  1. Gradient Checkpointing: Saves memory by recomputing activations during backward pass
  2. Activation Pooling: Reuses tensor memory to reduce garbage collection
  3. Model Sharding: Distributes layers across multiple GPUs

Example usage:

Constructors

Constructor Summary
TrainingMemoryManager(TrainingMemoryConfig config, IEnumerable<ILayer<T>> layers) Initializes a new instance of the TrainingMemoryManager.

Properties

Property Summary
ActivationPoolStats PoolStats { get; } Gets pool statistics if activation pooling is enabled.
TrainingMemoryConfig Config { get; } Gets the memory configuration.
bool IsCheckpointingEnabled { get; } Gets whether gradient checkpointing is enabled.
bool IsPoolingEnabled { get; } Gets whether activation pooling is enabled.
bool IsShardingEnabled { get; } Gets whether model sharding is enabled.

Methods

Method Summary
MemorySavingsEstimate EstimateMemorySavings(long modelParameters, int batchSize, int sequenceLength) Estimates memory savings from current configuration.
Tensor<T> BackwardSequence(IReadOnlyList<ILayer<T>> layers, Tensor<T> outputGradient) Performs backward pass through multiple layers with recomputation.
Tensor<T> ForwardSequence(IEnumerable<ILayer<T>> layers, Tensor<T> input) Performs forward pass through multiple layers with checkpointing.
Tensor<T> ForwardWithCheckpoint(ILayer<T> layer, Tensor<T> input, int layerIndex) Performs a forward pass with checkpointing for a single layer.
Tensor<T> RentTensor(int[] shape) Rents a tensor from the activation pool.
Tensor<T> ShardedForward(Tensor<T> input) Performs forward pass through sharded model.
bool ShouldCheckpoint(int layerIndex) Determines if a specific layer should be checkpointed.
long GetPoolMemoryUsage() Gets current memory usage from the activation pool.
void ClearCheckpoints() Clears all stored checkpoints to free memory.
void ComputeCheckpointIndices(ILayeredModel<T> layeredModel) Determines which layers should be checkpointed using layer-aware metadata from an ILayeredModel.
void ComputeCheckpointIndices(int totalLayers, IReadOnlyList<string> layerTypes) Determines which layers should be checkpointed based on configuration.
void Dispose() Disposes resources used by the memory manager.
void ReturnTensor(Tensor<T> tensor) Returns a tensor to the activation pool.