OptimizerMomentStorage
Enums · AiDotNet.Enums
How an Adam-family optimizer stores its first and second moments (m, v): the bulk of a large model’s training memory after the weights themselves.
For Beginners
AdamW remembers two numbers per weight. Storing them in fewer bits saves memory with
little or no change in how the model trains: BFloat16 halves that memory and Int8Blockwise
quarters it.
How It Works
The update math is fp32 in every mode; only the stored moments are narrower, as in bitsandbytes’ 8-bit optimizers. Device and host steps keep the same state, so a run can move between them and checkpoints stay in one format.
Fields
| Field | Summary |
|---|---|
OptimizerMomentStorage BFloat16 |
bfloat16 moments, round-to-nearest-even: 4 bytes per parameter. |
OptimizerMomentStorage Float32 |
32-bit moments: 8 bytes per parameter for m and v together. |
OptimizerMomentStorage Int8Blockwise |
8-bit block-wise dynamic quantization (Dettmers et al., “8-bit Optimizers via Block-wise Quantization”): one byte per moment element plus one scale per block, about 2 bytes per parameter. |
int value__ |