IAudioCodec<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for neural audio codecs that compress and decompress audio.

Known implementations: NeuralAudioCodecBase<T>

For Beginners

A neural audio codec is like an AI-powered audio compressor. It converts audio into a very compact code (tokens), then converts that code back to audio. Think of it like a zip file for audio, but using AI instead of traditional compression.

Two key operations:

  • Encode: Audio waveform -> compact tokens (small numbers)
  • Decode: Compact tokens -> reconstructed audio waveform

Uses:

  • Ultra-low bitrate audio streaming (1-6 kbps vs 128 kbps for MP3)
  • Audio tokens for AI language models
  • High-quality audio compression for storage

How It Works

Neural audio codecs use neural networks to compress audio into compact discrete tokens and reconstruct audio from those tokens. They achieve much higher compression ratios than traditional codecs (MP3, AAC) at comparable quality. The tokens can also serve as input to language models for audio generation.

Properties

Property Summary
int CodebookSize { get; } Gets the codebook size (vocabulary size per quantizer).
int NumQuantizers { get; } Gets the number of quantizer levels (codebooks).
int SampleRate { get; } Gets the sample rate of audio this codec operates on.
int TokenFrameRate { get; } Gets the frame rate of the encoded tokens (tokens per second).

Methods

Method Summary
Task<Tensor<T>> DecodeAsync(int[] tokens, CancellationToken cancellationToken) Decodes tokens asynchronously.
Task<int[]> EncodeAsync(Tensor<T> audio, CancellationToken cancellationToken) Encodes audio asynchronously.
Tensor<T> Decode(int[] tokens) Decodes tokens back into audio.
Tensor<T> DecodeEmbeddings(Tensor<T> embeddings) Reconstructs audio from continuous embeddings.
Tensor<T> EncodeEmbeddings(Tensor<T> audio) Encodes audio into continuous embeddings (before quantization).
double GetBitrate(int? numQuantizers) Gets the bitrate at the given number of quantizers in bits per second.
int[] Encode(Tensor<T> audio) Encodes audio into discrete tokens.