ImplicitPosteriorGenerator<T>

Models & Types · AiDotNet.MetaLearning.Components

SImPa’s IMPLICIT task-specific posterior: task parameters are produced by pushing latent noise through a generator network, w = G(z; lambda) with z ~ U[0,1]^d.

For Beginners

Instead of assuming the “uncertainty” about a task’s parameters is a simple bell curve, this feeds random numbers into a small network that outputs a full set of parameters. Run it again with different random numbers and you get a different plausible set. The collection of everything it can output IS the uncertainty, and it can be any shape at all.

How It Works

This is the contribution of Nguyen, Do and Carneiro, “PAC-Bayes meta-learning with implicit task-specific posteriors” (arXiv:2003.02455), which the paper’s abstract states as a “generative-based approach to estimate the posterior of task-specific model parameters more expressively compared to the usual assumption based on a multivariate normal distribution with a diagonal covariance matrix”.

WHY THIS CANNOT BE A GAUSSIAN, which is the whole point: a diagonal-covariance normal forces every parameter to vary independently and symmetrically about its mean, so it cannot represent a posterior with correlated parameters or several modes. Few-shot tasks routinely have several distinct plausible solutions, and a diagonal Gaussian must either straddle them — putting most of its mass where no solution lies — or collapse onto one. A generator has no such restriction: an arbitrarily shaped distribution is reachable by warping uniform noise, at the cost of having no density at all.

HAVING NO DENSITY IS NOT AN OVERSIGHT, it is the defining property. q(w) can be SAMPLED but never evaluated, so the KL term in the PAC-Bayes bound cannot be written in closed form and needs CompressionLemmaKLEstimator. Any implementation that reaches for a closed-form diagonal-Gaussian KL has, by that act, replaced this method with the baseline it was designed to beat.

Constructors

Constructor Summary
ImplicitPosteriorGenerator(int outputDimension, int latentDimension, int firstHiddenWidth, int secondHiddenWidth, Random rng) Creates a generator producing outputDimension-length parameter vectors.

Properties

Property Summary
int LatentDimension { get; } Gets the latent noise dimension.
int OutputDimension { get; } Gets the dimension of the generated parameter vector.
int ParameterCount { get; } Gets the number of generator weights in lambda.

Methods

Method Summary
IReadOnlyList<Vector<T>> SampleMany(int count, Random rng) Draws count independent samples — the particle set standing in for the posterior wherever an expectation under q is needed.
ImplicitPosteriorGenerator<T> Clone() Creates an independent generator with the same architecture and the same weights.
Vector<T> Generate(IReadOnlyList<double> latent) Deterministically maps a supplied latent vector to parameters, for tests that need to separate the generator’s mapping from the noise source.
Vector<T> GetParameters() Gets the flattened generator weights lambda.
Vector<T> Sample(Random rng) Draws one sample w = G(z; lambda) with a fresh z ~ U[0,1]^latent.
void SetParameters(Vector<T> lambda) Sets the flattened generator weights lambda.

Fields

Field Summary
int PaperFirstHiddenWidth The paper’s first hidden width.
int PaperLatentDimension The paper’s latent dimension: z ~ U[0,1]^128.
int PaperSecondHiddenWidth The paper’s second hidden width.