Adam8BitOptimizerOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.Models.Options

Configuration options for the 8-bit Adam optimization algorithm, which reduces memory usage by quantizing optimizer states.

Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → AdamOptimizerOptions<T, TInput, TOutput> → Adam8BitOptimizerOptions<T, TInput, TOutput>

For Beginners

Training large neural networks requires storing “optimizer state” - extra numbers for each parameter that help the optimizer make better updates. Standard Adam stores two numbers per parameter (momentum and variance), which can use a lot of memory for large models.

8-bit Adam compresses these numbers using a technique called quantization, similar to how JPEG compresses images. This reduces memory usage significantly with minimal impact on training quality. It’s especially useful when training large models where optimizer memory becomes a bottleneck.

How It Works

8-bit Adam provides the same optimization behavior as standard Adam but stores the first and second moment estimates (m and v) using 8-bit quantized representations instead of full precision floating point. This reduces memory usage by approximately 4x for these optimizer states, which is significant for large models.

Memory Savings Example: For a model with 1 billion parameters:

  • Standard Adam: 8 GB for optimizer states (2 states × 4 bytes × 1B params)
  • 8-bit Adam: ~2 GB for optimizer states (2 states × 1 byte × 1B params + scaling factors)

Constructors

Constructor Summary
Adam8BitOptimizerOptions() Initializes the options with their defaults.
Adam8BitOptimizerOptions(Adam8BitOptimizerOptions<T, TInput, TOutput> other) Copy constructor: copies every Adam and base-class setting (through the Adam copy constructor) and every 8-bit setting declared here, so a cloned configuration never falls back to a default.

Properties

Property Summary
bool CompressBothMoments { get; set; } Gets or sets whether to compress both first and second moments.
bool UseBFloat16MomentStorage { get; set; } Stores the optimizer moment state (m and v) as BFloat16 (2 bytes/element) instead of the default 8-bit block-quantized representation (1 byte/element).
bool UseDecoupledWeightDecay { get; set; } Gets or sets whether WeightDecay is applied as decoupled AdamW decay (true, default) rather than L2-coupled gradient decay (false).
bool UseDynamicQuantization { get; set; } Gets or sets whether to use dynamic quantization that adapts the scale during training.
bool UseStochasticRounding { get; set; } Gets or sets whether to use stochastic rounding during quantization.
double QuantizationPercentile { get; set; } Gets or sets the percentile of each block’s magnitudes used as its quantization scale.
double WeightDecay { get; set; } Gets or sets the weight decay coefficient applied during each optimizer step.
int BlockSize { get; set; } Gets or sets the block size for block-wise quantization.
int FullPrecisionUpdateFrequency { get; set; } Gets or sets the frequency of full-precision state updates.
int Min8BitSize { get; set; } Parameters with fewer elements than this keep full-precision Adam moments instead of 8-bit ones.