MixedPrecisionContext

Models & Types · AiDotNet.MixedPrecision

Manages master weights (FP32) and working weights (FP16) for mixed-precision training.

Implements: IDisposable

For Beginners

Mixed-precision training uses two copies of model parameters:

  1. Master Weights (FP32):
  • High-precision copy of all parameters
  • Used for parameter updates to maintain accuracy
  • Stored in memory but not used for forward/backward passes
  1. Working Weights (FP16):
  • Low-precision copy used for computation
  • Used in forward and backward passes
  • Faster and uses less memory
  • Synced from master weights before each forward pass

The workflow:

  1. Cast master weights (FP32) to working weights (FP16)
  2. Forward pass using FP16 weights → faster, less memory
  3. Backward pass in FP16 → computes FP16 gradients
  4. Cast gradients to FP32 and unscale
  5. Update master weights in FP32 → maintains precision
  6. Repeat from step 1

This approach combines the speed of FP16 with the numerical stability of FP32.

How It Works

Technical Details: The context maintains:

  • Dictionary mapping parameter names to FP32 master copies
  • Dictionary mapping parameter names to FP16 working copies
  • Synchronization methods to cast between precisions
  • Integration with LossScaler for gradient management

Constructors

Constructor Summary
MixedPrecisionContext(MixedPrecisionConfig config) Initializes a new mixed-precision training context.

Properties

Property Summary
IReadOnlyCollection<string> ParameterNames { get; } Gets the names of all parameters being managed.
LossScaler<float> LossScaler { get; } Loss scaler for gradient scaling and overflow detection.
MixedPrecisionConfig Config { get; } Configuration for mixed-precision training.
bool IsInitialized { get; set; } Whether the context has been initialized with parameters.
long ParameterCount { get; set; } Number of parameters managed by this context.

Methods

Method Summary
Vector<Half> GetWorkingWeights(string parameterName) Gets the working weights (FP16) for a parameter group.
Vector<float> CastWeightsToBF16(string parameterName) Round-trips the master weights through BF16 (truncate-low-16-bits-of-FP32) to emulate a forward pass that would have used BF16 working weights.
Vector<float> GetMasterWeights(string parameterName) Gets the master weights (FP32) for a parameter group.
bool EvictFp32Snapshot(Tensor<float> param) Evict the cached FP32 master snapshot for param.
bool HasFullFP32Precision(float masterWeight) Returns true if masterWeight carries low-mantissa bits that an FP16 / BF16 round-trip would have zeroed out — i.e.
bool PrepareGradientsForUpdate(Vector<Half> gradientsHalf, Vector<float> gradientsFloat) Converts FP16 gradients to FP32, unscales them, and checks for overflow.
string ToString() Gets a summary of the context’s current state.
void CastWeightsToFP16() Converts master weights (FP32) to working weights (FP16) for forward pass.
void Dispose() Disposes of the context and releases resources.
void Initialize(Dictionary<string, Vector<float>> namedParameters) Initializes the context with multiple named parameter groups.
void Initialize(Vector<float> parameters, string parameterName) Initializes the context with model parameters.
void Reset() Resets the context, clearing all weights and statistics.
void UpdateMasterWeights(Vector<float> gradients, float learningRate, string parameterName) Updates master weights with FP32 gradients after unscaling.