AudioLM<T>
Models & Types · AiDotNet.Audio.Generation
AudioLM: audio generation as language modeling over semantic and acoustic tokens (Borsos et al. 2022), in three stages — semantic, coarse acoustic and fine acoustic.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioLM<T>
Implements: IAudioGenerator<T>
For Beginners
give AudioLM three seconds of someone speaking and it continues in the same voice.
How It Works
Semantic tokens (w2v-BERT, 25 a second) carry the linguistic content and long-term structure; acoustic tokens (SoundStream, 50 frames of 12 levels a second) carry the voice, the recording conditions and the fine detail. The semantic stage continues the semantic tokens; the coarse stage predicts the first Q′ = 4 acoustic levels from all the semantic tokens; the fine stage predicts the remaining 8 from the coarse ones, 3 seconds at a time (§III-C).
Generation (§III-D): ContinueAudio continues a prompt recording (its semantic tokens come from the
injected W2vBert, or pass them to ContinueCodes); GenerateAcoustic
renders given semantic tokens (“acoustic generation”); GenerateUnconditional samples everything.
AudioLM has no text input, so the text-prompted IAudioGenerator methods throw; SPEAR-TTS adds text.
Training: Train takes the semantic tokens at 25 Hz as the input and the
recording’s codes [frames, Q] (or the recording) as the target, crops them as the paper does (30, 10 or
3 seconds) and updates the network CurrentStage names. See AudioLMOptions for the sizes
and for what is taken from lucidrains/audiolm-pytorch where the paper is silent.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var audioLm = new AudioLM<float>(architecture, new AudioLMOptions(), semanticTokenizer: w2vBert);
Tensor<float> continued = audioLm.ContinueAudio(threeSecondPrompt, extensionSeconds: 7);
Constructors
| Constructor | Summary |
|---|---|
AudioLM(NeuralNetworkArchitecture<T> architecture, AudioLMOptions options, W2vBert<T> semanticTokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates AudioLM. |
Properties
| Property | Summary |
|---|---|
AudioLMStage CurrentStage { get; set; } |
The network a training step updates. |
SoundStream<T> Codec { get; } |
The SoundStream codec whose codes AudioLM models. |
bool SupportsAudioContinuation { get; } |
|
bool SupportsAudioInpainting { get; } |
|
bool SupportsTextToAudio { get; } |
|
bool SupportsTextToMusic { get; } |
|
double FrameRate { get; } |
Codec frames per second (50). |
double MaxDurationSeconds { get; } |
Methods
| Method | Summary |
|---|---|
AudioGenerationOptions<T> GetDefaultOptions() |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(Tensor<T> input, Tensor<T> expectedOutput) |
The current stage’s loss on a sample, in inference mode with a fixed crop; for evaluation. |
Task<Tensor<T>> GenerateAudioAsync(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed, CancellationToken cancellationToken) |
|
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed) |
|
Tensor<T> Decode(int[] codes) |
The waveform of codes [Q, frames]. |
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
|
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
|
Tensor<T> GenerateUnconditional(double durationSeconds, int? seed) |
Unconditional generation (§III-D): samples semantic tokens from nothing, then renders them. |
Tensor<T> InpaintAudio(Tensor<T> audio, Tensor<T> mask, string prompt, int numInferenceSteps, int? seed) |
|
int[] ContinueCodes(int[] promptCodes, IReadOnlyList<int> promptSemantic, double extensionSeconds, int? seed) |
Continues a prompt given as codes [Q, P] and its semantic tokens (25 Hz) by extensionSeconds; returns the prompt’s and the continuation’s codes [Q, frames]. |
int[] GenerateAcoustic(IReadOnlyList<int> semantic, int? seed) |
Acoustic generation (§III-D): renders semantic tokens (25 Hz) to codes [Q, frames] with the coarse and fine stages. |