OptimizerMomentStorage

Enums · AiDotNet.Enums

How an Adam-family optimizer stores its first and second moments (m, v): the bulk of a large model’s training memory after the weights themselves.

For Beginners

AdamW remembers two numbers per weight. Storing them in fewer bits saves memory with little or no change in how the model trains: BFloat16 halves that memory and Int8Blockwise quarters it.

How It Works

The update math is fp32 in every mode; only the stored moments are narrower, as in bitsandbytes’ 8-bit optimizers. Device and host steps keep the same state, so a run can move between them and checkpoints stay in one format.

Fields

Field Summary
OptimizerMomentStorage BFloat16 bfloat16 moments, round-to-nearest-even: 4 bytes per parameter.
OptimizerMomentStorage Float32 32-bit moments: 8 bytes per parameter for m and v together.
OptimizerMomentStorage Int8Blockwise 8-bit block-wise dynamic quantization (Dettmers et al., “8-bit Optimizers via Block-wise Quantization”): one byte per moment element plus one scale per block, about 2 bytes per parameter.
int value__