SpeechTokenizer<T>
Models & Types · AiDotNet.Audio.Generation
SpeechTokenizer: EnCodec’s encoder–quantizer–decoder with a BiLSTM encoder, whose first residual codebook is distilled toward a self-supervised teacher’s representations so that it carries the content of speech and the remaining codebooks the paralinguistic detail.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NeuralAudioCodecBase<T> → SpeechTokenizer<T>
For Beginners
Give it speech and it returns eight rows of codes; the first row is close to the words, the rest to the voice. Decoding all eight rebuilds the speech.
How It Works
Reference: “SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models” (Zhang et al., ICLR 2024), with ZhangXInFD/SpeechTokenizer for what the paper leaves unstated.
The encoder and decoder (§3.1, App. D) are EnCodec’s SEANet (C = 32, strides 2, 4, 5, 8; residual kernels 3 and 1;
weight normalization; ELU) with the encoder’s two-layer LSTM replaced by a BiLSTM; at 16 kHz the 1024-dimensional
latent runs at 50 frames per second. Eight EMA codebooks of 1024 entries quantize it as EnCodec’s do. The first
codebook’s quantized output Q1 is distilled (§3.2) toward the teacher’s representations S (HuBERT’s average layer) by
the “D-axis” loss −(1/D) Σ_d log σ(cos(A Q1^(:,d), S^(:,d))), with A a learned projection.
Training (§3.3) adds a time L1 loss, EnCodec’s multi-scale mel loss, hinge adversarial and relative feature-matching losses against HiFi-Codec’s discriminators (MS-STFT, multi-period, multi-scale) and the commitment loss. The paper states no weights; they are the reference’s (time 500, mel 45, adversarial 1, feature 1, commitment 10, distillation 120). The teacher’s representations are an input: the reference also trains on representations extracted beforehand.
Constructors
| Constructor | Summary |
|---|---|
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Calls the fully specified overload with the omitted parameters’ defaults. |
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, SpeechTokenizerOptions options) |
Calls the fully specified overload with the omitted parameters’ defaults. |
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, SpeechTokenizerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable SpeechTokenizer. |
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, string modelPath) |
Calls the fully specified overload with the omitted parameters’ defaults. |
SpeechTokenizer(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeechTokenizerOptions options) |
Creates a trainable SpeechTokenizer. |
Properties
| Property | Summary |
|---|---|
int HopLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
double DistillationAgreement(Tensor<T> audio, Tensor<T> teacher) |
The mean over the teacher’s dimensions of the cosine similarity along time between the projected first-codebook output A·Q1 and the teacher’s representations — the agreement §3.2’s distillation raises. |
int[] EncodeSemantic(Tensor<T> audio) |
The semantic codes [frames] of speech: the first codebook’s, which carry the content. |
void LoadPretrainedWeights(string path) |
Loads the official SpeechTokenizer weights (the released SpeechTokenizer.pt, a PyTorch state dictionary of the reference model) into this model’s encoder, quantizer, decoder and semantic projection. |