DpSgdFusedStep<T>
Models & Types · AiDotNet.Training
Fused per-example DP-SGD (Abadi et al. 2016 §3, Algorithm 1). Runs each of K per-example forward+backward passes through the compiled plan (with the optimizer step’s learning rate set to zero so weights don’t drift between examples), extracts each example’s gradients from the parameter .Grad tensors, clips against the GLOBAL parameter-vector L2 norm, sums, adds a single Gaussian noise draw, averages, then applies the final aggregate update to the parameters.
Fused benefit over an eager per-example loop: each per-example
forward+backward runs the compiled plan (with GPU-resident parameters,
fused kernels, no per-op host↔device round-trip) instead of a fresh
non-persistent GradientTape. The clip / aggregate / noise
steps run in host code because their control flow (per-example L2 norm
then min-clip against C) doesn’t fit the current compiled-plan capture
model — but those are O(params) scalar operations, not the compute
bottleneck. The forward+backward per example IS the expensive part, and
that’s what gets fused.
Correctness contract: the clip-BEFORE-aggregate order is enforced by the class’s internal structure — callers can’t reverse it. This preserves the L2-sensitivity bound the Abadi 2016 privacy proof requires.
Implements: IDisposable
Constructors
| Constructor | Summary |
|---|---|
DpSgdFusedStep() |
Properties
| Property | Summary |
|---|---|
bool IsAvailable { get; } |
Whether the fused-resident DP-SGD path is available on this thread’s current engine. |
Methods
| Method | Summary |
|---|---|
bool TryStep(IReadOnlyList<Tensor<T>> parameters, Func<int, IReadOnlyList<Tensor<T>>> perExampleSlotData, Func<IReadOnlyList<Tensor<T>>, Tensor<T>> forward, Func<Tensor<T>, IReadOnlyList<Tensor<T>>, Tensor<T>> computeLoss, int batchSize, double clipNorm, double noiseMultiplier, Random rng, Dictionary<Tensor<T>, Tensor<T>> aggregatedGradients) |
Runs a full DP-SGD training step over batchSize examples. |
void Dispose() |
|
void Invalidate() |
Invalidates the cached compiled plan and persistent slots. |