DiffusionMemoryManager<T>

Models & Types · AiDotNet.Diffusion.Memory

Memory management utilities for diffusion models including gradient checkpointing, activation pooling, and model sharding integration.

For Beginners

Training large models uses a lot of memory because we need to store:

  1. The model parameters
  2. Intermediate activations (outputs from each layer during forward pass)
  3. Gradients for each parameter

This class helps reduce memory usage through several techniques:

Gradient Checkpointing:

  • Instead of storing all activations, only store “checkpoints”
  • During backward pass, recompute activations between checkpoints
  • Trades ~30% more compute time for ~50% less memory

Activation Pooling:

  • Reuse tensor memory instead of allocating new tensors
  • Reduces GC pressure and memory fragmentation

Model Sharding:

  • Split large models across multiple GPUs
  • Each GPU only holds part of the model

How It Works

This class provides memory-efficient training utilities specifically designed for large diffusion models (UNet, VAE, etc.) that may not fit in GPU memory during training.

Constructors

Constructor Summary
DiffusionMemoryManager(DiffusionMemoryConfig config, IEnumerable<ILayer<T>> layers) Initializes a new instance of the DiffusionMemoryManager class.

Properties

Property Summary
DiffusionMemoryConfig Config { get; } Memory configuration.
bool CheckpointingEnabled { get; } Whether gradient checkpointing is enabled.
bool PoolingEnabled { get; } Whether activation pooling is enabled.
bool ShardingEnabled { get; } Whether model sharding is active.

Methods

Method Summary
ActivationPoolStats GetPoolStats() Gets pooling statistics if available.
IReadOnlyDictionary<int, long> GetDeviceMemoryUsage() Gets memory usage per device.
MemoryEstimate EstimateMemory(int numLayers, long activationSizeBytes) Estimates memory savings from current configuration.
Tensor<T> RentTensor(int[] shape) Rents a tensor from the activation pool.
Tensor<T> ShardedForward(Tensor<T> input) Performs forward pass through sharded model.
Tensor<T> ShardedForward(Tensor<T> input, Tensor<T> context) Performs forward pass through sharded model with context.
ValueTuple<Tensor<T>, LayerCheckpointState<T>> ForwardWithCheckpointing(IReadOnlyList<ILayer<T>> layers, Tensor<T> input) Executes a forward pass through layers with optional checkpointing.
void ReturnTensor(Tensor<T> tensor) Returns a tensor to the activation pool for reuse.
void ShardedUpdateParameters(T learningRate) Updates parameters across all shards.