ResidualVectorQuantizerLayer<T>

Layers · AiDotNet.Audio.Codecs

Residual vector quantization (SoundStream, Zeghidour et al. 2021, Algorithm 1; EnCodec’s reference quantization/core_vq.py): each of n_q Euclidean codebooks quantizes the residual the previous ones left; the codebooks are updated by exponential moving averages of their assigned vectors (decay 0.99, Laplace smoothing 1e-5), initialized by k-means on the first training batch, and a code whose usage falls below the dead-code threshold is replaced by a random vector of the batch. Gradients pass straight through the quantization; the commitment loss is EnCodec’s Eq. 3, the sum over the codebooks used of the squared distance between each residual and its (stopped) code, or the reference code’s mean over codebooks of the mean squared error.

Inheritance: LayerBase<T> → ResidualVectorQuantizerLayer<T>

Implements: IResidualVectorQuantizer<T>, IShapeContract

How It Works

As the reference (and its issue #25, kept “for reproducibility”), each layer’s residual is reduced by the straight-through output, so gradient flows through the residual chain. The input and output are [1, dim, T].

Constructors

Constructor Summary
ResidualVectorQuantizerLayer(int dim, int quantizers, int bins, double decay, int kmeansIterations, int deadCodeThreshold, bool commitmentAsMean, bool expireCodesAfterEma) Creates the quantizer.

Properties

Property Summary
Tensor<T> CommitmentLoss { get; set; } The commitment loss of the last training forward over the codebooks used (a tape-connected scalar).
Tensor<T> FirstQuantized { get; set; } The first codebook’s quantized output [1, dim, T] of the last forward, straight-through to its input (SpeechTokenizer distills it toward a teacher).
bool CodebooksInitialized { get; }
bool SupportsTraining { get; }
int ActiveQuantizers { get; set; } The number of codebooks the forward uses (all by default; fewer for a lower bandwidth).
int Bins { get; } The codes per codebook.
int Quantizers { get; } The number of residual codebooks.
int[] LastCodes { get; } The codes [n_q, T] of the last forward (read from its one-hot codes when first asked for).

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
Tensor<T> Codebook(int q) The codebook [bins, dim] of quantizer q.
Tensor<T> Decode(int[] codes) The sum of the code vectors [1, dim, T] of codes [n_q, T].
int[] Encode(Tensor<T> x, int? quantizers) The codes [n_q, T] of [1, dim, T] (no codebook update).
void InitializeCodebooks(Tensor<T> latent)
void ResetState()