FactorizedVectorQuantizerLayer<T>

Layers · AiDotNet.Audio.Codecs

DAC’s residual vector quantizer with factorized, L2-normalized codes (Kumar et al. 2023, §3.2 and App. A; reference dac/nn/quantize.py): each codebook projects its residual to a low-dimensional space (W_in, a weight-normalized 1×1 convolution), looks up the nearest code by cosine similarity (both sides L2-normalized), and projects the code back (W_out); the codebooks are learned by gradient with the VQ-VAE codebook and commitment losses and the straight-through estimator — no EMA, k-means initialization or restarts.

Inheritance: LayerBase<T> → FactorizedVectorQuantizerLayer<T>

Implements: IResidualVectorQuantizer<T>, IShapeContract

How It Works

App. A states both losses on the L2-normalized vectors: L_VQ = ‖sg[ℓ2(z_proj)] − ℓ2(e_k)‖² + β‖ℓ2(z_proj) − sg[ℓ2(e_k)]‖²; the reference code takes them on the raw projections and codes (LossesOnRawVectors). Each is the mean squared error over the projected dimensions and frames, summed over the codebooks used.

Constructors

Constructor Summary
FactorizedVectorQuantizerLayer(int dim, int quantizers, int bins, int codeDim, bool lossesOnRawVectors, bool useL1Loss) Creates the quantizer.

Properties

Property Summary
Tensor<T> CodebookLoss { get; set; } The codebook loss of the last training forward, summed over the codebooks used.
Tensor<T> CommitmentLoss { get; set; }
bool CodebooksInitialized { get; }
bool LossesOnRawVectors { get; } Whether the losses compare raw vectors (the reference code) rather than L2-normalized ones.
bool SupportsTraining { get; }
int ActiveQuantizers { get; set; }
int[] LastCodes { get; } The codes [n_q, T] of the last forward (read from its one-hot codes when first asked for).

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
Tensor<T> Codebook(int q) Codebook q``[bins, codeDim].
Tensor<T> Decode(int[] codes)
int[] Encode(Tensor<T> latent, int? quantizers)
void InitializeCodebooks(Tensor<T> latent)
void ResetState()