CompressionLemmaKLEstimator<T>

Models & Types · AiDotNet.MetaLearning.Components

Estimates KL[q || p] from SAMPLES ONLY, via the compression lemma and a learned scalar phi-network — SImPa’s answer to a posterior that has no density.

For Beginners

KL divergence normally needs a formula for how likely each outcome is. Here we only have samples, no formula. So a small helper network is trained to tell the two sample sets apart as sharply as it can, and how well it manages is itself a measure of how different they are.

How It Works

Lemma 1 of Nguyen, Do and Carneiro (arXiv:2003.02455), the compression lemma: E_q[phi(w)] - ln E_p[e^phi(w)] <= KL[q || p], with equality attainable. So maximizing the left-hand side over a function class drives it UP toward the true KL, and its value at the maximum is the estimate. The paper’s Eq. 11: omega* = argmax_omega E_q[phi(w; omega)] - ln E_p[e^phi(w; omega)].

WHY A LEARNED ESTIMATOR IS NECESSARY rather than fussy. The task posterior is ImplicitPosteriorGenerator, which can be sampled but never evaluated, so ln q(w) does not exist to be plugged into the KL integral. Every closed form — the diagonal-Gaussian KL most of all — requires a density for BOTH arguments. This estimator needs only the ability to draw samples, which is exactly what an implicit distribution offers.

IT IS A LOWER BOUND, and that direction is the safe one for a generalization bound: the KL enters the PAC-Bayes bound positively, so under-estimating it makes the reported bound optimistic. That is a real caveat and is why the estimate is only meaningful after Maximize has been run — an untrained phi returns approximately zero for any pair of distributions, which would silently erase the KL term altogether.

Constructors

Constructor Summary
CompressionLemmaKLEstimator(int inputDimension, int hiddenWidth, Random rng) Creates the estimator.

Properties

Property Summary
int ParameterCount { get; } Gets the number of phi-network weights.

Methods

Method Summary
CompressionLemmaKLEstimator<T> Clone() Creates an independent estimator with the same architecture and the same weights.
Vector<T> GetParameters() Gets the flattened phi-network weights.
double EstimateKL(IReadOnlyList<Vector<T>> posteriorSamples, IReadOnlyList<Vector<T>> priorSamples, int steps, double learningRate, Random rng) Estimates KL[q || p]: maximizes the compression-lemma objective and floors the result at 0.
double Maximize(IReadOnlyList<Vector<T>> posteriorSamples, IReadOnlyList<Vector<T>> priorSamples, int steps, double learningRate, Random rng) Ascends the compression-lemma objective over omega and returns the resulting KL estimate.
double Objective(IReadOnlyList<Vector<T>> posteriorSamples, IReadOnlyList<Vector<T>> priorSamples) The compression-lemma objective E_q[phi] - ln E_p[e^phi], which lower-bounds KL[q || p].
double Phi(Vector<T> w) Evaluates the scalar phi(w; omega).
void SetParameters(Vector<T> omega) Sets the flattened phi-network weights.

Fields

Field Summary
int PaperMonteCarloSamples Monte Carlo samples the paper draws when training the phi-network per task.