TrainingMemoryConfig

Options & Configuration · AiDotNet.Training.Memory

Configuration for training memory management including gradient checkpointing, activation pooling, and model sharding.

For Beginners

Training neural networks requires a lot of memory:

  1. Model weights: The parameters being trained
  2. Activations: Intermediate results saved for backpropagation
  3. Gradients: Computed during backward pass

This configuration helps reduce memory usage through:

  • Gradient Checkpointing: Trade compute for memory by recomputing activations
  • Activation Pooling: Reuse memory buffers to reduce garbage collection
  • Model Sharding: Split large models across multiple GPUs

How It Works

Example usage with AiModelBuilder:

Constructors

Constructor Summary
TrainingMemoryConfig()

Properties

Property Summary
InitializationStrategyType DefaultInitializationStrategy { get; set; } Gets or sets the default initialization strategy type.
PoolingOptions TensorPoolOptions { get; set; } Gets or sets the tensor pooling options.
ShardingStrategy ShardingStrategy { get; set; } Gets or sets the sharding strategy.
WeightFileFormat WeightsFileFormat { get; set; } Gets or sets the format of the weights file.
bool CheckpointAttentionLayers { get; set; } Gets or sets whether to checkpoint attention layers specifically.
bool CheckpointResidualBlocks { get; set; } Gets or sets whether to checkpoint residual blocks.
bool TrackMemoryStatistics { get; set; } Gets or sets whether to track detailed memory statistics.
bool UseActivationPooling { get; set; } Gets or sets whether to use activation pooling.
bool UseGradientCheckpointing { get; set; } Gets or sets whether to use gradient checkpointing.
bool UsePipelineParallelism { get; set; } Gets or sets whether to use pipeline parallelism.
bool UseSharedTensorPool { get; set; } Gets or sets whether to use a shared global tensor pool.
bool UseTensorPooling { get; set; } Gets or sets whether tensor pooling is enabled.
double MemoryWarningThreshold { get; set; } Gets or sets the warning threshold for memory usage (as fraction of max).
int CheckpointEveryNLayers { get; set; } Gets or sets how often to create checkpoints (every N layers).
int MicroBatchCount { get; set; } Gets or sets the number of micro-batches for pipeline parallelism.
int NumDevices { get; set; } Gets or sets the number of devices for model sharding.
long HighActivationMemoryThresholdBytes { get; set; } Gets or sets the activation memory threshold (in bytes) above which a layer is automatically checkpointed regardless of other settings.
long MaxPoolMemoryMB { get; set; } Gets or sets the maximum memory for the activation pool in megabytes.
string WeightsFilePath { get; set; } Gets or sets the path to a weights file for FromFile initialization.

Methods

Method Summary
TrainingMemoryConfig AggressivePooling(int maxPoolSizeMB) Creates a configuration with aggressive memory pooling.
TrainingMemoryConfig Default() Creates a default configuration with minimal memory optimization.
TrainingMemoryConfig FastConstruction() Creates a configuration for fast model construction (good for testing).
TrainingMemoryConfig ForConvNets() Creates a configuration optimized for convolutional networks.
TrainingMemoryConfig ForTransferLearning(string weightsPath, WeightFileFormat format) Creates a configuration for transfer learning from a pre-trained model.
TrainingMemoryConfig ForTransformers() Creates a configuration optimized for transformer models.
TrainingMemoryConfig MemoryEfficient() Creates a memory-efficient configuration for large models.
TrainingMemoryConfig MultiGpu(int numDevices) Creates a configuration for multi-GPU training.
TrainingMemoryConfig SpeedOptimized() Creates a speed-optimized configuration (less memory optimization).