SImPaAlgorithm<T, TInput, TOutput>
Models & Types · AiDotNet.MetaLearning.Algorithms
SImPa — statistical implicit PAC-Bayes meta-learning: a PAC-Bayes meta-learner whose task-specific posterior is IMPLICIT, generated from noise rather than assumed to be a diagonal Gaussian.
Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → SImPaAlgorithm<T, TInput, TOutput>
Implements: IGeneratedParameterRegistrar<T>
For Beginners
Ordinary meta-learning gives one best guess of a task’s parameters. This keeps a whole distribution of plausible parameter sets — represented by a small network that turns random numbers into parameters, so the distribution can be any shape — and it comes with a mathematical guarantee about performance on tasks it has never seen.
How It Works
Cuong Nguyen, Thanh-Toan Do and Gustavo Carneiro, “PAC-Bayes meta-learning with implicit task-specific posteriors” (arXiv:2003.02455). The authors name the method SImPa.
THREE PARTS, each independently testable:
Posterior (ImplicitPosteriorGenerator) draws task parameters as
w = G(z; lambda) with z ~ U[0,1]^128; KLEstimator
(CompressionLemmaKLEstimator) recovers the KL term from SAMPLES via the compression
lemma, because an implicit posterior has no density to integrate; and
PacBayesMetaBound assembles the paper’s two-level bound over both unseen samples and
unseen tasks.
WHAT THIS REPLACED, and why it was not a small correction. The previous implementation of this
citation used a single POINT posterior with a closed-form diagonal-Gaussian KL
(0.5 * sum_d (theta_post_d - theta_prior_d)^2 / sigma_d^2), plus a “data-dependent prior” phase
split and a “flex” parameter interpolating toward ERM. None of that appears in the cited paper, and the
diagonal Gaussian is specifically the assumption the paper’s abstract names as the thing its
generative posterior is “more expressive” than. So the code implemented the baseline the paper was
written to beat, under the paper’s name.
Example
var model = new SimpleRegression<double>();
var options = new SImPaOptions<double, Matrix<double>, Vector<double>>(model);
var simpa = new SImPaAlgorithm<double, Matrix<double>, Vector<double>>(options);
var taskBatch = new TaskBatch<double, Matrix<double>, Vector<double>>(
new IMetaLearningTask<double, Matrix<double>, Vector<double>>[]
{
new MetaLearningTask<double, Matrix<double>, Vector<double>>()
});
double bound = Convert.ToDouble(simpa.MetaTrain(taskBatch));
Constructors
| Constructor | Summary |
|---|---|
SImPaAlgorithm(SImPaOptions<T, TInput, TOutput> options) |
Creates the meta-learner. |
Properties
| Property | Summary |
|---|---|
CompressionLemmaKLEstimator<T> KLEstimator { get; } |
Gets the compression-lemma KL estimator. |
ImplicitPosteriorGenerator<T> Posterior { get; } |
Gets the implicit task-specific posterior generator. |
MetaLearningAlgorithmType AlgorithmType { get; } |
|
double LastBound { get; set; } |
Gets the most recent PAC-Bayes bound value computed by MetaTrain, or NaN before the first call. |
double LastTaskKL { get; set; } |
Gets the most recent task-level KL estimate, or NaN before the first call. |
Methods
| Method | Summary |
|---|---|
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) |
|
IReadOnlyList<Vector<T>> SamplePosterior(IMetaLearningTask<T, TInput, TOutput> task, int? sampleCount) |
Adapts to a task and returns the individual posterior PARTICLES rather than their mean. |
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) |