DpSgdFusedStep<T>

Models & Types · AiDotNet.Training

Fused per-example DP-SGD (Abadi et al. 2016 §3, Algorithm 1). Runs each of K per-example forward+backward passes through the compiled plan (with the optimizer step’s learning rate set to zero so weights don’t drift between examples), extracts each example’s gradients from the parameter .Grad tensors, clips against the GLOBAL parameter-vector L2 norm, sums, adds a single Gaussian noise draw, averages, then applies the final aggregate update to the parameters.

Fused benefit over an eager per-example loop: each per-example forward+backward runs the compiled plan (with GPU-resident parameters, fused kernels, no per-op host↔device round-trip) instead of a fresh non-persistent GradientTape. The clip / aggregate / noise steps run in host code because their control flow (per-example L2 norm then min-clip against C) doesn’t fit the current compiled-plan capture model — but those are O(params) scalar operations, not the compute bottleneck. The forward+backward per example IS the expensive part, and that’s what gets fused.

Correctness contract: the clip-BEFORE-aggregate order is enforced by the class’s internal structure — callers can’t reverse it. This preserves the L2-sensitivity bound the Abadi 2016 privacy proof requires.

Implements: IDisposable

Constructors

Constructor Summary
DpSgdFusedStep()

Properties

Property Summary
bool IsAvailable { get; } Whether the fused-resident DP-SGD path is available on this thread’s current engine.

Methods

Method Summary
bool TryStep(IReadOnlyList<Tensor<T>> parameters, Func<int, IReadOnlyList<Tensor<T>>> perExampleSlotData, Func<IReadOnlyList<Tensor<T>>, Tensor<T>> forward, Func<Tensor<T>, IReadOnlyList<Tensor<T>>, Tensor<T>> computeLoss, int batchSize, double clipNorm, double noiseMultiplier, Random rng, Dictionary<Tensor<T>, Tensor<T>> aggregatedGradients) Runs a full DP-SGD training step over batchSize examples.
void Dispose()
void Invalidate() Invalidates the cached compiled plan and persistent slots.