SpeechTokenizer<T>

Models & Types · AiDotNet.Audio.Generation

SpeechTokenizer: EnCodec’s encoder–quantizer–decoder with a BiLSTM encoder, whose first residual codebook is distilled toward a self-supervised teacher’s representations so that it carries the content of speech and the remaining codebooks the paralinguistic detail.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NeuralAudioCodecBase<T> → SpeechTokenizer<T>

For Beginners

Give it speech and it returns eight rows of codes; the first row is close to the words, the rest to the voice. Decoding all eight rebuilds the speech.

How It Works

Reference: “SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models” (Zhang et al., ICLR 2024), with ZhangXInFD/SpeechTokenizer for what the paper leaves unstated.

The encoder and decoder (§3.1, App. D) are EnCodec’s SEANet (C = 32, strides 2, 4, 5, 8; residual kernels 3 and 1; weight normalization; ELU) with the encoder’s two-layer LSTM replaced by a BiLSTM; at 16 kHz the 1024-dimensional latent runs at 50 frames per second. Eight EMA codebooks of 1024 entries quantize it as EnCodec’s do. The first codebook’s quantized output Q1 is distilled (§3.2) toward the teacher’s representations S (HuBERT’s average layer) by the “D-axis” loss −(1/D) Σ_d log σ(cos(A Q1^(:,d), S^(:,d))), with A a learned projection.

Training (§3.3) adds a time L1 loss, EnCodec’s multi-scale mel loss, hinge adversarial and relative feature-matching losses against HiFi-Codec’s discriminators (MS-STFT, multi-period, multi-scale) and the commitment loss. The paper states no weights; they are the reference’s (time 500, mel 45, adversarial 1, feature 1, commitment 10, distillation 120). The teacher’s representations are an input: the reference also trains on representations extracted beforehand.

Constructors

Constructor Summary
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Calls the fully specified overload with the omitted parameters’ defaults.
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, SpeechTokenizerOptions options) Calls the fully specified overload with the omitted parameters’ defaults.
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, SpeechTokenizerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable SpeechTokenizer.
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, string modelPath) Calls the fully specified overload with the omitted parameters’ defaults.
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeechTokenizerOptions options) Creates a trainable SpeechTokenizer.

Properties

Property Summary
int HopLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
double DistillationAgreement(Tensor<T> audio, Tensor<T> teacher) The mean over the teacher’s dimensions of the cosine similarity along time between the projected first-codebook output A·Q1 and the teacher’s representations — the agreement §3.2’s distillation raises.
int[] EncodeSemantic(Tensor<T> audio) The semantic codes [frames] of speech: the first codebook’s, which carry the content.
void LoadPretrainedWeights(string path) Loads the official SpeechTokenizer weights (the released SpeechTokenizer.pt, a PyTorch state dictionary of the reference model) into this model’s encoder, quantizer, decoder and semantic projection.