AudioLM<T>

Models & Types · AiDotNet.Audio.Generation

AudioLM: audio generation as language modeling over semantic and acoustic tokens (Borsos et al. 2022), in three stages — semantic, coarse acoustic and fine acoustic.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioLM<T>

Implements: IAudioGenerator<T>

For Beginners

give AudioLM three seconds of someone speaking and it continues in the same voice.

How It Works

Semantic tokens (w2v-BERT, 25 a second) carry the linguistic content and long-term structure; acoustic tokens (SoundStream, 50 frames of 12 levels a second) carry the voice, the recording conditions and the fine detail. The semantic stage continues the semantic tokens; the coarse stage predicts the first Q′ = 4 acoustic levels from all the semantic tokens; the fine stage predicts the remaining 8 from the coarse ones, 3 seconds at a time (§III-C).

Generation (§III-D): ContinueAudio continues a prompt recording (its semantic tokens come from the injected W2vBert, or pass them to ContinueCodes); GenerateAcoustic renders given semantic tokens (“acoustic generation”); GenerateUnconditional samples everything. AudioLM has no text input, so the text-prompted IAudioGenerator methods throw; SPEAR-TTS adds text.

Training: Train takes the semantic tokens at 25 Hz as the input and the recording’s codes [frames, Q] (or the recording) as the target, crops them as the paper does (30, 10 or 3 seconds) and updates the network CurrentStage names. See AudioLMOptions for the sizes and for what is taken from lucidrains/audiolm-pytorch where the paper is silent.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
    NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var audioLm = new AudioLM<float>(architecture, new AudioLMOptions(), semanticTokenizer: w2vBert);
Tensor<float> continued = audioLm.ContinueAudio(threeSecondPrompt, extensionSeconds: 7);

Constructors

Constructor Summary
AudioLM(NeuralNetworkArchitecture<T> architecture, AudioLMOptions options, W2vBert<T> semanticTokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates AudioLM.

Properties

Property Summary
AudioLMStage CurrentStage { get; set; } The network a training step updates.
SoundStream<T> Codec { get; } The SoundStream codec whose codes AudioLM models.
bool SupportsAudioContinuation { get; }
bool SupportsAudioInpainting { get; }
bool SupportsTextToAudio { get; }
bool SupportsTextToMusic { get; }
double FrameRate { get; } Codec frames per second (50).
double MaxDurationSeconds { get; }

Methods

Method Summary
AudioGenerationOptions<T> GetDefaultOptions()
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(Tensor<T> input, Tensor<T> expectedOutput) The current stage’s loss on a sample, in inference mode with a fixed crop; for evaluation.
Task<Tensor<T>> GenerateAudioAsync(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed, CancellationToken cancellationToken)
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed)
Tensor<T> Decode(int[] codes) The waveform of codes [Q, frames].
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed)
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed)
Tensor<T> GenerateUnconditional(double durationSeconds, int? seed) Unconditional generation (§III-D): samples semantic tokens from nothing, then renders them.
Tensor<T> InpaintAudio(Tensor<T> audio, Tensor<T> mask, string prompt, int numInferenceSteps, int? seed)
int[] ContinueCodes(int[] promptCodes, IReadOnlyList<int> promptSemantic, double extensionSeconds, int? seed) Continues a prompt given as codes [Q, P] and its semantic tokens (25 Hz) by extensionSeconds; returns the prompt’s and the continuation’s codes [Q, frames].
int[] GenerateAcoustic(IReadOnlyList<int> semantic, int? seed) Acoustic generation (§III-D): renders semantic tokens (25 Hz) to codes [Q, frames] with the coarse and fine stages.