DAC<T>

Models & Types · AiDotNet.Audio.Effects

DAC, the Descript Audio Codec: a Snake-activated convolutional encoder–decoder with a residual vector quantizer of factorized, L2-normalized codes, trained with quantizer dropout against multi-period and multi-band multi-scale STFT discriminators and a multi-scale mel loss.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NeuralAudioCodecBase<T> → DAC<T>

For Beginners

DAC compresses any kind of audio — speech, music, sound effects — into about 86 frames of codes per second and reconstructs it with very little loss; using fewer codebooks lowers the bitrate.

How It Works

Reference: “High-Fidelity Audio Compression with Improved RVQGAN” (Kumar et al., 2023), with descriptinc/descript-audio-codec for what the paper leaves unstated.

The encoder and decoder (§3.1, §4.3) are SoundStream’s fully convolutional design with Snake activations: strides (2, 4, 8, 8) to a 1024-dimensional latent (hop 512, about 86 frames per second at 44.1 kHz) and rates (8, 8, 4, 2) from a 1536-wide decoder to a tanh waveform. The quantizer (§3.2, App. A) projects each residual to 8 dimensions, picks the nearest of 1024 codes by cosine similarity, and projects back; its codebooks learn by the VQ-VAE codebook and commitment losses. Quantizer dropout (§3.3) draws n_q ~ U[1, 9] for each example.

Training (§3.4–3.5, §4.3) minimizes 15 · the L1 distance of log-mel spectrograms over seven windows (32 … 2048 samples, 5 … 320 mel bins) + 2 · feature matching + 1 · the hinge adversarial loss + 1 · codebook + 0.25 · commitment, against five period discriminators (2, 3, 5, 7, 11) and complex STFT discriminators at windows 2048, 1024 and 512 split into five frequency bands. AdamW (1e-4, β = 0.8, 0.9) trains both, its rate decaying by 0.999996 each step.

Constructors

Constructor Summary
DAC(NeuralNetworkArchitecture<T> architecture, DACOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable DAC.
DAC(NeuralNetworkArchitecture<T> architecture, string modelPath, DACOptions options) Creates a trainable DAC.

Properties

Property Summary
int HopLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
void LoadDescriptCheckpoint(string path) Loads descript-audio-codec’s .pth checkpoint (its state_dict entry, or a bare state dict) in descript’s layout; see LoadDescriptWeights.
void LoadPretrainedWeights(string path) Loads the official DAC weights from a safetensors file in the Hugging Face layout (for example descript/dac_44khz’s model.safetensors) into this model’s encoder, quantizer and decoder.