NeuralAudioCodecBase<T>
Base Classes · AiDotNet.Audio.Codecs
The shared pipeline of the neural audio codecs (SoundStream, EnCodec, DAC, SpeechTokenizer): an encoder to a latent sequence, a residual vector quantizer whose number of codebooks sets the bandwidth, and a decoder back to audio, trained adversarially on random segments with one bandwidth drawn per step.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NeuralAudioCodecBase<T>
Implements: IAudioCodec<T>
Known derived types: DAC<T>, EnCodec<T>, SoundStream<T>, SpeechTokenizer<T>
How It Works
A model supplies its encoder, decoder and quantizer (CreateCodec), its discriminators, its generator
and discriminator objectives and its optimizers. The base owns segment and bandwidth sampling, the alternating
discriminator and generator steps (each updating only its own layers), the IAudioCodec surface and
a deterministic reconstruction objective for evaluation.
Each step trains the discriminator first (with DiscriminatorUpdateProbability) on a generation
computed without updating the codebooks, then the generator; the generation is the same, as the weights have not
changed in between.
Properties
| Property | Summary |
|---|---|
bool CodebooksInitialized { get; } |
Whether the quantizer’s codebooks have been fitted (by training or a load). |
int CodebookSize { get; } |
|
int HopLength { get; } |
Samples per latent frame. |
int NumQuantizers { get; } |
|
int TokenFrameRate { get; } |
Methods
| Method | Summary |
|---|---|
Task<Tensor<T>> DecodeAsync(int[] tokens, CancellationToken cancellationToken) |
|
Task<int[]> EncodeAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> Decode(int[] tokens) |
|
Tensor<T> DecodeEmbeddings(Tensor<T> embeddings) |
|
Tensor<T> EncodeEmbeddings(Tensor<T> audio) |
|
double GetBitrate(int? numQuantizers) |
|
int QuantizersForBandwidth(double kbps) |
The number of codebooks for a bandwidth in kbps (each carries frame rate × log2(bins) bits per second). |
int[] Encode(Tensor<T> audio) |
|
void InitializeCodebooks(Tensor<T> audio) |
Fits the codebooks that have never been fitted from audio’s latents, by the data-dependent initialization the first training batch would run (k-means for SoundStream/EnCodec); fitted codebooks are left as they are. |
void Train(Tensor<T> input, Tensor<T> expectedOutput) |