ImplicitPosteriorGenerator<T>
Models & Types · AiDotNet.MetaLearning.Components
SImPa’s IMPLICIT task-specific posterior: task parameters are produced by pushing latent noise
through a generator network, w = G(z; lambda) with z ~ U[0,1]^d.
For Beginners
Instead of assuming the “uncertainty” about a task’s parameters is a simple bell curve, this feeds random numbers into a small network that outputs a full set of parameters. Run it again with different random numbers and you get a different plausible set. The collection of everything it can output IS the uncertainty, and it can be any shape at all.
How It Works
This is the contribution of Nguyen, Do and Carneiro, “PAC-Bayes meta-learning with implicit task-specific posteriors” (arXiv:2003.02455), which the paper’s abstract states as a “generative-based approach to estimate the posterior of task-specific model parameters more expressively compared to the usual assumption based on a multivariate normal distribution with a diagonal covariance matrix”.
WHY THIS CANNOT BE A GAUSSIAN, which is the whole point: a diagonal-covariance normal forces every parameter to vary independently and symmetrically about its mean, so it cannot represent a posterior with correlated parameters or several modes. Few-shot tasks routinely have several distinct plausible solutions, and a diagonal Gaussian must either straddle them — putting most of its mass where no solution lies — or collapse onto one. A generator has no such restriction: an arbitrarily shaped distribution is reachable by warping uniform noise, at the cost of having no density at all.
HAVING NO DENSITY IS NOT AN OVERSIGHT, it is the defining property. q(w) can be SAMPLED but
never evaluated, so the KL term in the PAC-Bayes bound cannot be written in closed form and needs
CompressionLemmaKLEstimator. Any implementation that reaches for a closed-form
diagonal-Gaussian KL has, by that act, replaced this method with the baseline it was designed to beat.
Constructors
| Constructor | Summary |
|---|---|
ImplicitPosteriorGenerator(int outputDimension, int latentDimension, int firstHiddenWidth, int secondHiddenWidth, Random rng) |
Creates a generator producing outputDimension-length parameter vectors. |
Properties
| Property | Summary |
|---|---|
int LatentDimension { get; } |
Gets the latent noise dimension. |
int OutputDimension { get; } |
Gets the dimension of the generated parameter vector. |
int ParameterCount { get; } |
Gets the number of generator weights in lambda. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<Vector<T>> SampleMany(int count, Random rng) |
Draws count independent samples — the particle set standing in for the posterior wherever an expectation under q is needed. |
ImplicitPosteriorGenerator<T> Clone() |
Creates an independent generator with the same architecture and the same weights. |
Vector<T> Generate(IReadOnlyList<double> latent) |
Deterministically maps a supplied latent vector to parameters, for tests that need to separate the generator’s mapping from the noise source. |
Vector<T> GetParameters() |
Gets the flattened generator weights lambda. |
Vector<T> Sample(Random rng) |
Draws one sample w = G(z; lambda) with a fresh z ~ U[0,1]^latent. |
void SetParameters(Vector<T> lambda) |
Sets the flattened generator weights lambda. |
Fields
| Field | Summary |
|---|---|
int PaperFirstHiddenWidth |
The paper’s first hidden width. |
int PaperLatentDimension |
The paper’s latent dimension: z ~ U[0,1]^128. |
int PaperSecondHiddenWidth |
The paper’s second hidden width. |