SImPaAlgorithm<T, TInput, TOutput>

Models & Types · AiDotNet.MetaLearning.Algorithms

SImPa — statistical implicit PAC-Bayes meta-learning: a PAC-Bayes meta-learner whose task-specific posterior is IMPLICIT, generated from noise rather than assumed to be a diagonal Gaussian.

Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → SImPaAlgorithm<T, TInput, TOutput>

Implements: IGeneratedParameterRegistrar<T>

For Beginners

Ordinary meta-learning gives one best guess of a task’s parameters. This keeps a whole distribution of plausible parameter sets — represented by a small network that turns random numbers into parameters, so the distribution can be any shape — and it comes with a mathematical guarantee about performance on tasks it has never seen.

How It Works

Cuong Nguyen, Thanh-Toan Do and Gustavo Carneiro, “PAC-Bayes meta-learning with implicit task-specific posteriors” (arXiv:2003.02455). The authors name the method SImPa.

THREE PARTS, each independently testable: Posterior (ImplicitPosteriorGenerator) draws task parameters as w = G(z; lambda) with z ~ U[0,1]^128; KLEstimator (CompressionLemmaKLEstimator) recovers the KL term from SAMPLES via the compression lemma, because an implicit posterior has no density to integrate; and PacBayesMetaBound assembles the paper’s two-level bound over both unseen samples and unseen tasks.

WHAT THIS REPLACED, and why it was not a small correction. The previous implementation of this citation used a single POINT posterior with a closed-form diagonal-Gaussian KL (0.5 * sum_d (theta_post_d - theta_prior_d)^2 / sigma_d^2), plus a “data-dependent prior” phase split and a “flex” parameter interpolating toward ERM. None of that appears in the cited paper, and the diagonal Gaussian is specifically the assumption the paper’s abstract names as the thing its generative posterior is “more expressive” than. So the code implemented the baseline the paper was written to beat, under the paper’s name.

Example

var model = new SimpleRegression<double>();
var options = new SImPaOptions<double, Matrix<double>, Vector<double>>(model);
var simpa = new SImPaAlgorithm<double, Matrix<double>, Vector<double>>(options);
var taskBatch = new TaskBatch<double, Matrix<double>, Vector<double>>(
    new IMetaLearningTask<double, Matrix<double>, Vector<double>>[]
    {
        new MetaLearningTask<double, Matrix<double>, Vector<double>>()
    });
double bound = Convert.ToDouble(simpa.MetaTrain(taskBatch));

Constructors

Constructor Summary
SImPaAlgorithm(SImPaOptions<T, TInput, TOutput> options) Creates the meta-learner.

Properties

Property Summary
CompressionLemmaKLEstimator<T> KLEstimator { get; } Gets the compression-lemma KL estimator.
ImplicitPosteriorGenerator<T> Posterior { get; } Gets the implicit task-specific posterior generator.
MetaLearningAlgorithmType AlgorithmType { get; }
double LastBound { get; set; } Gets the most recent PAC-Bayes bound value computed by MetaTrain, or NaN before the first call.
double LastTaskKL { get; set; } Gets the most recent task-level KL estimate, or NaN before the first call.

Methods

Method Summary
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task)
IReadOnlyList<Vector<T>> SamplePosterior(IMetaLearningTask<T, TInput, TOutput> task, int? sampleCount) Adapts to a task and returns the individual posterior PARTICLES rather than their mean.
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch)