MixedPrecisionContext
Models & Types · AiDotNet.MixedPrecision
Manages master weights (FP32) and working weights (FP16) for mixed-precision training.
Implements: IDisposable
For Beginners
Mixed-precision training uses two copies of model parameters:
- Master Weights (FP32):
- High-precision copy of all parameters
- Used for parameter updates to maintain accuracy
- Stored in memory but not used for forward/backward passes
- Working Weights (FP16):
- Low-precision copy used for computation
- Used in forward and backward passes
- Faster and uses less memory
- Synced from master weights before each forward pass
The workflow:
- Cast master weights (FP32) to working weights (FP16)
- Forward pass using FP16 weights → faster, less memory
- Backward pass in FP16 → computes FP16 gradients
- Cast gradients to FP32 and unscale
- Update master weights in FP32 → maintains precision
- Repeat from step 1
This approach combines the speed of FP16 with the numerical stability of FP32.
How It Works
Technical Details: The context maintains:
- Dictionary mapping parameter names to FP32 master copies
- Dictionary mapping parameter names to FP16 working copies
- Synchronization methods to cast between precisions
- Integration with LossScaler for gradient management
Constructors
| Constructor | Summary |
|---|---|
MixedPrecisionContext(MixedPrecisionConfig config) |
Initializes a new mixed-precision training context. |
Properties
| Property | Summary |
|---|---|
IReadOnlyCollection<string> ParameterNames { get; } |
Gets the names of all parameters being managed. |
LossScaler<float> LossScaler { get; } |
Loss scaler for gradient scaling and overflow detection. |
MixedPrecisionConfig Config { get; } |
Configuration for mixed-precision training. |
bool IsInitialized { get; set; } |
Whether the context has been initialized with parameters. |
long ParameterCount { get; set; } |
Number of parameters managed by this context. |
Methods
| Method | Summary |
|---|---|
Vector<Half> GetWorkingWeights(string parameterName) |
Gets the working weights (FP16) for a parameter group. |
Vector<float> CastWeightsToBF16(string parameterName) |
Round-trips the master weights through BF16 (truncate-low-16-bits-of-FP32) to emulate a forward pass that would have used BF16 working weights. |
Vector<float> GetMasterWeights(string parameterName) |
Gets the master weights (FP32) for a parameter group. |
bool EvictFp32Snapshot(Tensor<float> param) |
Evict the cached FP32 master snapshot for param. |
bool HasFullFP32Precision(float masterWeight) |
Returns true if masterWeight carries low-mantissa bits that an FP16 / BF16 round-trip would have zeroed out — i.e. |
bool PrepareGradientsForUpdate(Vector<Half> gradientsHalf, Vector<float> gradientsFloat) |
Converts FP16 gradients to FP32, unscales them, and checks for overflow. |
string ToString() |
Gets a summary of the context’s current state. |
void CastWeightsToFP16() |
Converts master weights (FP32) to working weights (FP16) for forward pass. |
void Dispose() |
Disposes of the context and releases resources. |
void Initialize(Dictionary<string, Vector<float>> namedParameters) |
Initializes the context with multiple named parameter groups. |
void Initialize(Vector<float> parameters, string parameterName) |
Initializes the context with model parameters. |
void Reset() |
Resets the context, clearing all weights and statistics. |
void UpdateMasterWeights(Vector<float> gradients, float learningRate, string parameterName) |
Updates master weights with FP32 gradients after unscaling. |