TrainingDiagnosticsConfig
Options & Configuration · AiDotNet.Configuration
Process-global control for training-pipeline diagnostic output
(gradient norms, optimizer step traces, tape events). Mirrors the
three-orthogonal-knobs design of GpuDiagnosticsConfig:
environment variable, static configuration, and custom sink.
How It Works
AiDotNet’s training pipeline (NeuralNetworkBase.TrainWithTape)
can fail in subtle ways — wrong-direction gradient flow, dropped
layer gradients, optimizer skipping parameters, fused-path bailouts.
These bugs are hard to diagnose without per-parameter gradient
visibility (see github.com/ooples/AiDotNet#1328 for an example where
the model trained but only the bias of the final dense head got
useful gradients). This class provides production-grade hooks so
regression suites and end-user debugging can introspect training
without modifying the library.
Three orthogonal controls (all work simultaneously):
Example
var target = new Vector<double>(new double[] { 0.0, 1.0, 0.0, 1.0 });
var input = new Matrix<double>(new double[,] { { 1.0, 2.0 }, { 3.0, 4.0 }, { 5.0, 6.0 }, { 7.0, 8.0 } });
// Option A: environment variable (set before process start)
// AIDOTNET_TRAINING_DIAGNOSTICS=perstep
// Option B: static config
AiDotNet.Configuration.TrainingDiagnosticsConfig.Level =
TrainingDiagnosticLevel.PerStep;
// Option C: custom sink that collects GradientNormEvents for assertions
var grads = new System.Collections.Generic.List<GradientNormEvent>();
AiDotNet.Configuration.TrainingDiagnosticsConfig.Sink = evt => {
if (evt is GradientNormEvent g) grads.Add(g);
};
AiDotNet.Configuration.TrainingDiagnosticsConfig.Level =
TrainingDiagnosticLevel.PerStep;
var result = new AiModelBuilder<double, Matrix<double>, Vector<double>>()
.ConfigureModel(new ARModel<double>())
.Build(input, target);
AiDotNet.Configuration.TrainingDiagnosticsConfig.Level =
TrainingDiagnosticLevel.Silent;
// grads now has one record per trainable parameter for that step.
Properties
| Property | Summary |
|---|---|
TrainingDiagnosticLevel Level { get; set; } |
Current verbosity level. |
TrainingDiagnosticSink Sink { get; set; } |
Optional sink that receives diagnostic events when set. |
bool PerStepEnabled { get; set; } |
Convenience boolean for the most common case (“turn on per-step gradient diagnostics for one test”). |
int CurrentStepIndex { get; } |
Monotonically incrementing step counter, advanced once per call to TrainWithTape. |
int FusedOptimizerHits { get; } |
How many training steps ran on the fused compiled path since the last ResetFusedOptimizerCounters. |
int FusedOptimizerMisses { get; } |
How many training steps fell back to the eager tape since the last ResetFusedOptimizerCounters. |
string LastFusedOptimizerMissReason { get; } |
Why the most recent fallback happened, or null if the fused path has not missed. |
Methods
| Method | Summary |
|---|---|
void Emit(TrainingDiagnosticEvent evt) |
Emits a structured diagnostic event respecting the current Level. |
void EmitMessage(TrainingDiagnosticLevel level, string message) |
Convenience helper for emitting a free-form message at a specific level. |
void ResetFusedOptimizerCounters() |
Resets the fused-path hit/miss counters and the last miss reason. |
void ResetStepCounter() |