CenteringMechanism<T>

Models & Types · AiDotNet.SelfSupervisedLearning

Centering mechanism for preventing collapse in self-distillation methods.

Inheritance: ModelBase<T, Tensor<T>, Tensor<T>> → CenteringMechanism<T>

For Beginners

Centering is a crucial technique in DINO and similar methods that prevents the teacher network from collapsing to a trivial solution where it outputs the same constant for all inputs.

How It Works

How it works:

Why it prevents collapse: Without centering, the teacher could learn to output a constant vector for all inputs (trivial solution). By subtracting the running mean, we ensure the outputs are zero-centered on average, forcing the network to produce varied outputs.

Reference: Caron et al., “Emerging Properties in Self-Supervised Vision Transformers” (ICCV 2021)

Constructors

Constructor Summary
CenteringMechanism(int dimension, double momentum) Initializes a new instance of the CenteringMechanism class.

Properties

Property Summary
ILossFunction<T> DefaultLossFunction { get; }
double Momentum { get; } Gets the momentum for EMA updates.
int Dimension { get; } Gets the dimension of the center vector.

Methods

Method Summary
IFullModel<T, Tensor<T>, Tensor<T>> WithParameters(Vector<T> parameters)
T CenterNorm() Computes the L2 norm of the center (useful for monitoring).
T[] GetCenter() Gets the current center values.
Tensor<T> ApplyCenter(Tensor<T> input) Applies centering to the input tensor.
Tensor<T> CenterAndUpdate(Tensor<T> teacherOutput) Applies centering and updates in one step (common usage pattern).
Tensor<T> Predict(Tensor<T> input)
ValueTuple<T, T, T, T> CenterStatistics() Computes statistics about the center (useful for debugging).
void Reset() Resets the center to zeros.
void SetCenter(T[] center) Sets the center values directly.
void Train(Tensor<T> input, Tensor<T> expectedOutput)
void Update(Tensor<T> batchOutput) Updates the center using EMA with the given batch.
void UpdateFromMultiple(IList<Tensor<T>> outputs) Updates the center using multiple batches of outputs.