VarianceAdaptorLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
The variance adaptor of FastSpeech 2: duration prediction and length regulation, then pitch and energy prediction, each added to the expanded sequence as a quantized embedding.
Inheritance: LayerBase<T> → VarianceAdaptorLayer<T>
Implements: IShapeContract
How It Works
Ren et al. 2021, §2.3 and Fig. 1b. In order:
Bin ranges, which the paper leaves to the implementation: pitch bins span WORLD DIO’s search range (71 to 800 Hz
by default) and energy bins span the range the STFT energy can take for audio in [-1, 1]
(MaxStftEnergy), so every value the extractors produce has a bin. Reference implementations fit
both ranges from the training set instead; SetEnergyRange does that for energy.
As a layer, the forward pass runs the inference path, so a model whose layer stack is
encoder → adaptor → decoder synthesizes with an ordinary sequential forward. Training uses
Adapt, which takes the ground-truth targets and returns the predictions for the losses.
Constructors
| Constructor | Summary |
|---|---|
VarianceAdaptorLayer(int hiddenSize, int filterSize, int kernelSize, double dropoutRate, int pitchBins, double pitchMinHz, double pitchMaxHz, int energyBins, double energyMin, double energyMax, bool usePitch, bool useEnergy) |
Creates a variance adaptor (defaults: FastSpeech 2, App. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
bool UseEnergy { get; } |
Whether the energy branch is present. |
bool UsePitch { get; } |
Whether the pitch branch is present. |
double DropoutRate { get; } |
Dropout of each predictor. |
double DurationScale { get; set; } |
Speed control for inference: predicted durations are multiplied by this before rounding (FastSpeech’s α, Ren et al. |
double EnergyMax { get; } |
Highest energy bin edge. |
double EnergyMin { get; } |
Lowest energy bin edge. |
double PitchMaxHz { get; } |
Highest pitch bin edge, in Hz. |
double PitchMinHz { get; } |
Lowest pitch bin edge, in Hz. |
int EnergyBins { get; } |
Number of energy bins. |
int FilterSize { get; } |
Convolution channels of each predictor. |
int HiddenSize { get; } |
Width of the hidden sequence. |
int KernelSize { get; } |
Convolution kernel of each predictor. |
int PitchBins { get; } |
Number of pitch bins. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
Tensor<T> AddFrameVariance(Tensor<T> frames, Tensor<T> pitch, Tensor<T> energy) |
Adds the pitch and energy embeddings to a frame-level hidden sequence, from the given frame pitch and energy, or from the adaptor’s predictions where those are null. |
Tensor<T> LengthRegulate(Tensor<T> phonemes, int[] durations) |
FastSpeech’s length regulator: repeats row i of phonemes``durations[i] times (Expand). |
VarianceAdaptation<T> Adapt(Tensor<T> hidden, VarianceTargets<T> targets, double durationScale) |
Runs the adaptor. |
double MaxStftEnergy(int fftSize) |
The largest L2 norm a frame of a periodic-Hann STFT of size fftSize can have for audio in [-1, 1]: by Parseval the half spectrum carries at most N·Σw²/2 + (Σw)² = N²·7/16 of energy. |
void ResetState() |
|
void SetEnergyRange(double min, double max) |
Sets the energy bin range, normally the minimum and maximum frame energy of the training data. |