TacotronSpectrogram

Models & Types · AiDotNet.TextToSpeech

The mel spectrogram and frame energy that Tacotron 2 and the acoustic models following it are trained on.

How It Works

Tacotron 2 (Shen et al. 2018) defines the target most neural TTS models reuse — FastSpeech 2 states it follows Shen et al. with a 1024-sample frame and 256-sample hop at 22050 Hz (Ren et al. 2021, §3.1). Its reference implementation (TacotronSTFT) takes the magnitude of a centred, reflect-padded, Hann-windowed STFT, applies librosa’s mel filterbank (Slaney mel scale, Slaney area normalization), and compresses with log(max(x, 1e-5)). FastSpeech 2’s energy is the L2 norm of each frame of that magnitude.

The filterbank follows librosa filters.mel exactly, so targets match those produced by the published preprocessing.

Constructors

Constructor Summary
TacotronSpectrogram(int sampleRate, int fftSize, int hopLength, int windowLength, int melChannels, double minFrequency, double maxFrequency, double clipValue) Creates the transform.

Properties

Property Summary
double ClipValue { get; } Floor applied before the log.
double FramePeriodMs { get; } Frame period in milliseconds (hop / sample rate), the period pitch must be extracted at to align.
double[] MelBasis { get; } The mel filterbank, [melChannels, fftSize/2 + 1] (librosa filters.mel, Slaney).
int FftSize { get; } FFT size.
int HopLength { get; } Hop between frames, in samples.
int MelChannels { get; } Number of mel channels.
int SampleRate { get; } Sampling rate, in Hz.
int WindowLength { get; } Analysis window length, in samples.

Methods

Method Summary
Tensor<T> Energy(Tensor<T> magnitude) FastSpeech 2’s frame energy: the L2 norm of each STFT magnitude frame, [frames].
Tensor<T> LogMel(Tensor<T> audio) The log mel spectrogram of a recording, [frames, melChannels].
Tensor<T> LogMelOfMagnitude(Tensor<T> magnitude) The log mel spectrogram log(max(M S, clip)) of an STFT magnitude [frames, fftSize/2 + 1], [frames, melChannels].
Tensor<T> Magnitude(Tensor<T> audio) The STFT magnitude, [frames, fftSize/2 + 1]: the signal reflect-padded by fftSize / 2 on each side, framed at the hop, windowed by the centred periodic Hann window and transformed (librosa stft(center=True, pad_mode='reflect'), Tacotron…
int FrameCount(int sampleCount) Number of frames for a signal of sampleCount samples: the reference count 1 + (samples + 2 (fftSize / 2) − fftSize) / hop of the reflect-padded STFT (1 + samples / hop for an even FFT size).