CompressionLemmaKLEstimator<T>
Models & Types · AiDotNet.MetaLearning.Components
Estimates KL[q || p] from SAMPLES ONLY, via the compression lemma and a learned scalar
phi-network — SImPa’s answer to a posterior that has no density.
For Beginners
KL divergence normally needs a formula for how likely each outcome is. Here we only have samples, no formula. So a small helper network is trained to tell the two sample sets apart as sharply as it can, and how well it manages is itself a measure of how different they are.
How It Works
Lemma 1 of Nguyen, Do and Carneiro (arXiv:2003.02455), the compression lemma:
E_q[phi(w)] - ln E_p[e^phi(w)] <= KL[q || p], with equality attainable. So maximizing the
left-hand side over a function class drives it UP toward the true KL, and its value at the maximum is
the estimate. The paper’s Eq. 11:
omega* = argmax_omega E_q[phi(w; omega)] - ln E_p[e^phi(w; omega)].
WHY A LEARNED ESTIMATOR IS NECESSARY rather than fussy. The task posterior is
ImplicitPosteriorGenerator, which can be sampled but never evaluated, so
ln q(w) does not exist to be plugged into the KL integral. Every closed form — the
diagonal-Gaussian KL most of all — requires a density for BOTH arguments. This estimator needs only
the ability to draw samples, which is exactly what an implicit distribution offers.
IT IS A LOWER BOUND, and that direction is the safe one for a generalization bound: the KL enters the
PAC-Bayes bound positively, so under-estimating it makes the reported bound optimistic. That is a real
caveat and is why the estimate is only meaningful after Maximize has been run — an
untrained phi returns approximately zero for any pair of distributions, which would silently
erase the KL term altogether.
Constructors
| Constructor | Summary |
|---|---|
CompressionLemmaKLEstimator(int inputDimension, int hiddenWidth, Random rng) |
Creates the estimator. |
Properties
| Property | Summary |
|---|---|
int ParameterCount { get; } |
Gets the number of phi-network weights. |
Methods
| Method | Summary |
|---|---|
CompressionLemmaKLEstimator<T> Clone() |
Creates an independent estimator with the same architecture and the same weights. |
Vector<T> GetParameters() |
Gets the flattened phi-network weights. |
double EstimateKL(IReadOnlyList<Vector<T>> posteriorSamples, IReadOnlyList<Vector<T>> priorSamples, int steps, double learningRate, Random rng) |
Estimates KL[q || p]: maximizes the compression-lemma objective and floors the result at 0. |
double Maximize(IReadOnlyList<Vector<T>> posteriorSamples, IReadOnlyList<Vector<T>> priorSamples, int steps, double learningRate, Random rng) |
Ascends the compression-lemma objective over omega and returns the resulting KL estimate. |
double Objective(IReadOnlyList<Vector<T>> posteriorSamples, IReadOnlyList<Vector<T>> priorSamples) |
The compression-lemma objective E_q[phi] - ln E_p[e^phi], which lower-bounds KL[q || p]. |
double Phi(Vector<T> w) |
Evaluates the scalar phi(w; omega). |
void SetParameters(Vector<T> omega) |
Sets the flattened phi-network weights. |
Fields
| Field | Summary |
|---|---|
int PaperMonteCarloSamples |
Monte Carlo samples the paper draws when training the phi-network per task. |