TrainingDiagnosticsConfig

Options & Configuration · AiDotNet.Configuration

Process-global control for training-pipeline diagnostic output (gradient norms, optimizer step traces, tape events). Mirrors the three-orthogonal-knobs design of GpuDiagnosticsConfig: environment variable, static configuration, and custom sink.

How It Works

AiDotNet’s training pipeline (NeuralNetworkBase.TrainWithTape) can fail in subtle ways — wrong-direction gradient flow, dropped layer gradients, optimizer skipping parameters, fused-path bailouts. These bugs are hard to diagnose without per-parameter gradient visibility (see github.com/ooples/AiDotNet#1328 for an example where the model trained but only the bias of the final dense head got useful gradients). This class provides production-grade hooks so regression suites and end-user debugging can introspect training without modifying the library.

Three orthogonal controls (all work simultaneously):

Example

var target = new Vector<double>(new double[] { 0.0, 1.0, 0.0, 1.0 });
var input = new Matrix<double>(new double[,] { { 1.0, 2.0 }, { 3.0, 4.0 }, { 5.0, 6.0 }, { 7.0, 8.0 } });
// Option A: environment variable (set before process start)
//   AIDOTNET_TRAINING_DIAGNOSTICS=perstep
            
// Option B: static config
AiDotNet.Configuration.TrainingDiagnosticsConfig.Level =
    TrainingDiagnosticLevel.PerStep;
            
// Option C: custom sink that collects GradientNormEvents for assertions
var grads = new System.Collections.Generic.List<GradientNormEvent>();
AiDotNet.Configuration.TrainingDiagnosticsConfig.Sink = evt => {
    if (evt is GradientNormEvent g) grads.Add(g);
};
AiDotNet.Configuration.TrainingDiagnosticsConfig.Level =
    TrainingDiagnosticLevel.PerStep;
var result = new AiModelBuilder<double, Matrix<double>, Vector<double>>()
    .ConfigureModel(new ARModel<double>())
    .Build(input, target);
AiDotNet.Configuration.TrainingDiagnosticsConfig.Level =
    TrainingDiagnosticLevel.Silent;
// grads now has one record per trainable parameter for that step.

Properties

Property Summary
TrainingDiagnosticLevel Level { get; set; } Current verbosity level.
TrainingDiagnosticSink Sink { get; set; } Optional sink that receives diagnostic events when set.
bool PerStepEnabled { get; set; } Convenience boolean for the most common case (“turn on per-step gradient diagnostics for one test”).
int CurrentStepIndex { get; } Monotonically incrementing step counter, advanced once per call to TrainWithTape.
int FusedOptimizerHits { get; } How many training steps ran on the fused compiled path since the last ResetFusedOptimizerCounters.
int FusedOptimizerMisses { get; } How many training steps fell back to the eager tape since the last ResetFusedOptimizerCounters.
string LastFusedOptimizerMissReason { get; } Why the most recent fallback happened, or null if the fused path has not missed.

Methods

Method Summary
void Emit(TrainingDiagnosticEvent evt) Emits a structured diagnostic event respecting the current Level.
void EmitMessage(TrainingDiagnosticLevel level, string message) Convenience helper for emitting a free-form message at a specific level.
void ResetFusedOptimizerCounters() Resets the fused-path hit/miss counters and the last miss reason.
void ResetStepCounter()