EnCodec<T>

Models & Types · AiDotNet.Audio.Generation

EnCodec: a streaming convolutional encoder–decoder with an LSTM, residual vector quantization of its latent, and adversarial training against a multi-scale STFT discriminator with a loss balancer.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NeuralAudioCodecBase<T> → EnCodec<T>

For Beginners

EnCodec compresses audio into a few numbers per 13 ms and rebuilds the audio from them. More codebooks per frame means a higher bitrate and better quality.

How It Works

Reference: “High Fidelity Neural Audio Compression” (Défossez et al., 2022), with facebookresearch/encodec (the released model) and facebookresearch/audiocraft (the authors’ training code) for what the paper leaves unstated.

The encoder (§3.1) is a k = 7 convolution to C = 32 channels, four blocks of a residual unit and a strided convolution (kernel 2S; strides 2, 4, 5, 8) doubling the channels, a two-layer LSTM and a k = 7 convolution to D = 128; ELU throughout; the decoder mirrors it with transposed convolutions. The streamable model pads causally and uses weight normalization; the non-streamable one splits the padding and layer-normalizes. The latent is quantized by up to 32 residual codebooks of 1024 entries (§3.2) with EMA updates (decay 0.99), dead-code replacement and a straight-through gradient; a bandwidth (1.5 … 24 kbps) is drawn per step and selects the codebooks and a dedicated discriminator.

Training (§3.4) balances, on the output x̂, the time L1 loss, the multi-scale mel L1 + L2 loss (scales 2⁵ … 2¹¹, 64 bins), the hinge adversarial loss and the relative feature-matching loss of the MS-STFT discriminator with weights 0.1, 1, 3, 3 (Eq. 5, R = 1, β = 0.999), and adds the commitment loss outside the balancer. The discriminator is updated with probability 2/3 with the hinge loss; Adam (3e-4, β = 0.5, 0.9) trains both (§4.3).

Constructors

Constructor Summary
EnCodec(NeuralNetworkArchitecture<T> architecture, EnCodecOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable EnCodec.
EnCodec(NeuralNetworkArchitecture<T> architecture, string modelPath, EnCodecOptions options) Creates a trainable EnCodec.

Properties

Property Summary
int HopLength { get; }

Methods

Method Summary
IReadOnlyList<EncodedFrame<T>> EncodeFrames(Tensor<T> audio) Encodes audio chunk by chunk (§3.1 “Non-streamable”: 1 s chunks with a 10 ms overlap, each normalized), or as one chunk when no chunk length is configured.
ModelMetadata<T> GetModelMetadata()
Tensor<T> Decode(int[] tokens)
Tensor<T> DecodeFrames(IReadOnlyList<EncodedFrame<T>> frames) Decodes chunks and joins them by linear overlap-add with triangular weights (reference _linear_overlap_add).
int[] Encode(Tensor<T> audio)
void LoadAuthorWeights(string path, string prefix) Loads the original weight-normalized SEANet state dictionary (as safetensors or a supported PyTorch archive).
void LoadPretrainedWeights(string path) Loads the official EnCodec weights from a safetensors file in the Hugging Face layout (for example facebook/encodec_24khz’s model.safetensors) into this model’s encoder, quantizer and decoder.