FeatureWiseTransformation<T>
Models & Types · AiDotNet.MetaLearning.Algorithms
The feature-wise transformation layer of Tseng et al. (arXiv:2001.08735): a per-channel affine perturbation whose scale and bias are SAMPLED from learned hyper-parameters.
For Beginners
This jiggles the numbers coming out of the feature extractor — multiplying each channel by something near 1 and adding something near 0. Because the jiggle is different every time, whatever comes next cannot rely on the exact values, only on their pattern. How big the jiggle is allowed to be is the thing being learned.
How It Works
Given an activation z with C channels, the modulated activation is
Three details carry the method, and each is easy to get subtly wrong:
Constructors
| Constructor | Summary |
|---|---|
FeatureWiseTransformation(int featureDimension, double initialScale, double initialBias, Random random) |
Initializes the transformation with the paper’s pre-determined hyper-parameter values. |
Properties
| Property | Summary |
|---|---|
Vector<T> BiasHyperparameters { get; set; } |
theta_beta — one entry per channel, the bias term’s pre-softplus spread. |
Vector<T> ScaleHyperparameters { get; set; } |
theta_gamma — one entry per channel, the scale term’s pre-softplus spread. |
int FeatureDimension { get; } |
Gets the channel count C the transformation is defined over. |
Methods
| Method | Summary |
|---|---|
FeatureWiseTransformation<T> Clone() |
Creates an independent transformation holding copies of the same hyper-parameters. |
Vector<T> Apply(Vector<T> features) |
Applies one freshly-sampled transformation to a feature vector. |
double EffectiveBiasStdDev(int channel) |
The standard deviation actually used for channel channel’s bias term, i.e. |
double EffectiveScaleStdDev(int channel) |
The standard deviation actually used for channel channel’s scale term, i.e. |
void SetHyperparameters(Vector<T> scale, Vector<T> bias) |
Replaces both hyper-parameter vectors, for the learning-to-learn update. |