VarianceAdaptorLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

The variance adaptor of FastSpeech 2: duration prediction and length regulation, then pitch and energy prediction, each added to the expanded sequence as a quantized embedding.

Inheritance: LayerBase<T> → VarianceAdaptorLayer<T>

Implements: IShapeContract

How It Works

Ren et al. 2021, §2.3 and Fig. 1b. In order:

Bin ranges, which the paper leaves to the implementation: pitch bins span WORLD DIO’s search range (71 to 800 Hz by default) and energy bins span the range the STFT energy can take for audio in [-1, 1] (MaxStftEnergy), so every value the extractors produce has a bin. Reference implementations fit both ranges from the training set instead; SetEnergyRange does that for energy.

As a layer, the forward pass runs the inference path, so a model whose layer stack is encoder → adaptor → decoder synthesizes with an ordinary sequential forward. Training uses Adapt, which takes the ground-truth targets and returns the predictions for the losses.

Constructors

Constructor Summary
VarianceAdaptorLayer(int hiddenSize, int filterSize, int kernelSize, double dropoutRate, int pitchBins, double pitchMinHz, double pitchMaxHz, int energyBins, double energyMin, double energyMax, bool usePitch, bool useEnergy) Creates a variance adaptor (defaults: FastSpeech 2, App.

Properties

Property Summary
bool SupportsTraining { get; }
bool UseEnergy { get; } Whether the energy branch is present.
bool UsePitch { get; } Whether the pitch branch is present.
double DropoutRate { get; } Dropout of each predictor.
double DurationScale { get; set; } Speed control for inference: predicted durations are multiplied by this before rounding (FastSpeech’s α, Ren et al.
double EnergyMax { get; } Highest energy bin edge.
double EnergyMin { get; } Lowest energy bin edge.
double PitchMaxHz { get; } Highest pitch bin edge, in Hz.
double PitchMinHz { get; } Lowest pitch bin edge, in Hz.
int EnergyBins { get; } Number of energy bins.
int FilterSize { get; } Convolution channels of each predictor.
int HiddenSize { get; } Width of the hidden sequence.
int KernelSize { get; } Convolution kernel of each predictor.
int PitchBins { get; } Number of pitch bins.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
Tensor<T> AddFrameVariance(Tensor<T> frames, Tensor<T> pitch, Tensor<T> energy) Adds the pitch and energy embeddings to a frame-level hidden sequence, from the given frame pitch and energy, or from the adaptor’s predictions where those are null.
Tensor<T> LengthRegulate(Tensor<T> phonemes, int[] durations) FastSpeech’s length regulator: repeats row i of phonemes``durations[i] times (Expand).
VarianceAdaptation<T> Adapt(Tensor<T> hidden, VarianceTargets<T> targets, double durationScale) Runs the adaptor.
double MaxStftEnergy(int fftSize) The largest L2 norm a frame of a periodic-Hann STFT of size fftSize can have for audio in [-1, 1]: by Parseval the half spectrum carries at most N·Σw²/2 + (Σw)² = N²·7/16 of energy.
void ResetState()
void SetEnergyRange(double min, double max) Sets the energy bin range, normally the minimum and maximum frame energy of the training data.