TacotronSpectrogram
Models & Types · AiDotNet.TextToSpeech
The mel spectrogram and frame energy that Tacotron 2 and the acoustic models following it are trained on.
How It Works
Tacotron 2 (Shen et al. 2018) defines the target most neural TTS models reuse — FastSpeech 2 states it follows
Shen et al. with a 1024-sample frame and 256-sample hop at 22050 Hz (Ren et al. 2021, §3.1). Its reference
implementation (TacotronSTFT) takes the magnitude of a centred, reflect-padded, Hann-windowed STFT,
applies librosa’s mel filterbank (Slaney mel scale, Slaney area normalization), and compresses with
log(max(x, 1e-5)). FastSpeech 2’s energy is the L2 norm of each frame of that magnitude.
The filterbank follows librosa filters.mel exactly, so targets match those produced by the
published preprocessing.
Constructors
| Constructor | Summary |
|---|---|
TacotronSpectrogram(int sampleRate, int fftSize, int hopLength, int windowLength, int melChannels, double minFrequency, double maxFrequency, double clipValue) |
Creates the transform. |
Properties
| Property | Summary |
|---|---|
double ClipValue { get; } |
Floor applied before the log. |
double FramePeriodMs { get; } |
Frame period in milliseconds (hop / sample rate), the period pitch must be extracted at to align. |
double[] MelBasis { get; } |
The mel filterbank, [melChannels, fftSize/2 + 1] (librosa filters.mel, Slaney). |
int FftSize { get; } |
FFT size. |
int HopLength { get; } |
Hop between frames, in samples. |
int MelChannels { get; } |
Number of mel channels. |
int SampleRate { get; } |
Sampling rate, in Hz. |
int WindowLength { get; } |
Analysis window length, in samples. |
Methods
| Method | Summary |
|---|---|
Tensor<T> Energy(Tensor<T> magnitude) |
FastSpeech 2’s frame energy: the L2 norm of each STFT magnitude frame, [frames]. |
Tensor<T> LogMel(Tensor<T> audio) |
The log mel spectrogram of a recording, [frames, melChannels]. |
Tensor<T> LogMelOfMagnitude(Tensor<T> magnitude) |
The log mel spectrogram log(max(M S, clip)) of an STFT magnitude [frames, fftSize/2 + 1], [frames, melChannels]. |
Tensor<T> Magnitude(Tensor<T> audio) |
The STFT magnitude, [frames, fftSize/2 + 1]: the signal reflect-padded by fftSize / 2 on each side, framed at the hop, windowed by the centred periodic Hann window and transformed (librosa stft(center=True, pad_mode='reflect'), Tacotron… |
int FrameCount(int sampleCount) |
Number of frames for a signal of sampleCount samples: the reference count 1 + (samples + 2 (fftSize / 2) − fftSize) / hop of the reflect-padded STFT (1 + samples / hop for an even FFT size). |