TrainingMemoryConfig
Options & Configuration · AiDotNet.Training.Memory
Configuration for training memory management including gradient checkpointing, activation pooling, and model sharding.
For Beginners
Training neural networks requires a lot of memory:
- Model weights: The parameters being trained
- Activations: Intermediate results saved for backpropagation
- Gradients: Computed during backward pass
This configuration helps reduce memory usage through:
- Gradient Checkpointing: Trade compute for memory by recomputing activations
- Activation Pooling: Reuse memory buffers to reduce garbage collection
- Model Sharding: Split large models across multiple GPUs
How It Works
Example usage with AiModelBuilder:
Constructors
| Constructor | Summary |
|---|---|
TrainingMemoryConfig() |
Properties
| Property | Summary |
|---|---|
InitializationStrategyType DefaultInitializationStrategy { get; set; } |
Gets or sets the default initialization strategy type. |
PoolingOptions TensorPoolOptions { get; set; } |
Gets or sets the tensor pooling options. |
ShardingStrategy ShardingStrategy { get; set; } |
Gets or sets the sharding strategy. |
WeightFileFormat WeightsFileFormat { get; set; } |
Gets or sets the format of the weights file. |
bool CheckpointAttentionLayers { get; set; } |
Gets or sets whether to checkpoint attention layers specifically. |
bool CheckpointResidualBlocks { get; set; } |
Gets or sets whether to checkpoint residual blocks. |
bool TrackMemoryStatistics { get; set; } |
Gets or sets whether to track detailed memory statistics. |
bool UseActivationPooling { get; set; } |
Gets or sets whether to use activation pooling. |
bool UseGradientCheckpointing { get; set; } |
Gets or sets whether to use gradient checkpointing. |
bool UsePipelineParallelism { get; set; } |
Gets or sets whether to use pipeline parallelism. |
bool UseSharedTensorPool { get; set; } |
Gets or sets whether to use a shared global tensor pool. |
bool UseTensorPooling { get; set; } |
Gets or sets whether tensor pooling is enabled. |
double MemoryWarningThreshold { get; set; } |
Gets or sets the warning threshold for memory usage (as fraction of max). |
int CheckpointEveryNLayers { get; set; } |
Gets or sets how often to create checkpoints (every N layers). |
int MicroBatchCount { get; set; } |
Gets or sets the number of micro-batches for pipeline parallelism. |
int NumDevices { get; set; } |
Gets or sets the number of devices for model sharding. |
long HighActivationMemoryThresholdBytes { get; set; } |
Gets or sets the activation memory threshold (in bytes) above which a layer is automatically checkpointed regardless of other settings. |
long MaxPoolMemoryMB { get; set; } |
Gets or sets the maximum memory for the activation pool in megabytes. |
string WeightsFilePath { get; set; } |
Gets or sets the path to a weights file for FromFile initialization. |
Methods
| Method | Summary |
|---|---|
TrainingMemoryConfig AggressivePooling(int maxPoolSizeMB) |
Creates a configuration with aggressive memory pooling. |
TrainingMemoryConfig Default() |
Creates a default configuration with minimal memory optimization. |
TrainingMemoryConfig FastConstruction() |
Creates a configuration for fast model construction (good for testing). |
TrainingMemoryConfig ForConvNets() |
Creates a configuration optimized for convolutional networks. |
TrainingMemoryConfig ForTransferLearning(string weightsPath, WeightFileFormat format) |
Creates a configuration for transfer learning from a pre-trained model. |
TrainingMemoryConfig ForTransformers() |
Creates a configuration optimized for transformer models. |
TrainingMemoryConfig MemoryEfficient() |
Creates a memory-efficient configuration for large models. |
TrainingMemoryConfig MultiGpu(int numDevices) |
Creates a configuration for multi-GPU training. |
TrainingMemoryConfig SpeedOptimized() |
Creates a speed-optimized configuration (less memory optimization). |