CenteringMechanism<T>
Models & Types · AiDotNet.SelfSupervisedLearning
Centering mechanism for preventing collapse in self-distillation methods.
Inheritance: ModelBase<T, Tensor<T>, Tensor<T>> → CenteringMechanism<T>
For Beginners
Centering is a crucial technique in DINO and similar methods that prevents the teacher network from collapsing to a trivial solution where it outputs the same constant for all inputs.
How It Works
How it works:
Why it prevents collapse: Without centering, the teacher could learn to output a constant vector for all inputs (trivial solution). By subtracting the running mean, we ensure the outputs are zero-centered on average, forcing the network to produce varied outputs.
Reference: Caron et al., “Emerging Properties in Self-Supervised Vision Transformers” (ICCV 2021)
Constructors
| Constructor | Summary |
|---|---|
CenteringMechanism(int dimension, double momentum) |
Initializes a new instance of the CenteringMechanism class. |
Properties
| Property | Summary |
|---|---|
ILossFunction<T> DefaultLossFunction { get; } |
|
double Momentum { get; } |
Gets the momentum for EMA updates. |
int Dimension { get; } |
Gets the dimension of the center vector. |
Methods
| Method | Summary |
|---|---|
IFullModel<T, Tensor<T>, Tensor<T>> WithParameters(Vector<T> parameters) |
|
T CenterNorm() |
Computes the L2 norm of the center (useful for monitoring). |
T[] GetCenter() |
Gets the current center values. |
Tensor<T> ApplyCenter(Tensor<T> input) |
Applies centering to the input tensor. |
Tensor<T> CenterAndUpdate(Tensor<T> teacherOutput) |
Applies centering and updates in one step (common usage pattern). |
Tensor<T> Predict(Tensor<T> input) |
|
ValueTuple<T, T, T, T> CenterStatistics() |
Computes statistics about the center (useful for debugging). |
void Reset() |
Resets the center to zeros. |
void SetCenter(T[] center) |
Sets the center values directly. |
void Train(Tensor<T> input, Tensor<T> expectedOutput) |
|
void Update(Tensor<T> batchOutput) |
Updates the center using EMA with the given batch. |
void UpdateFromMultiple(IList<Tensor<T>> outputs) |
Updates the center using multiple batches of outputs. |