MusicGenModel<T>

Models & Types · AiDotNet.Audio.MusicGen

MusicGen (Copet et al. 2023): text-to-music generation with one transformer language model over the four codebooks of a 32 kHz EnCodec, interleaved by the “delay” pattern and conditioned on frozen T5 text embeddings through cross-attention, sampled with classifier-free guidance.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → MusicGenModel<T>

Implements: IAudioGenerator<T>, IPaperOptimizerVariant

For Beginners

Describe the music you want and MusicGen writes it: it predicts compressed audio a frame at a time, steered toward the description, and a codec turns those frames into sound.

How It Works

The text is tokenized with t5-base’s SentencePiece model and encoded by a T5 encoder (frozen, as the paper’s pretrained conditioner), projected to the model width. The language model (MusicGenCore) reads the delay-interleaved codes and is trained with each codebook’s cross-entropy over its valid steps. Training applies the paper’s condition merging, description dropout, word dropout and condition dropout; the optimizer is AdamW (or D-Adaptation for the 300M model) with clipping and a cosine schedule, and an EMA of the weights is what generation uses once training has updated it. Generation fills the pattern step by step with uncond + 3 · (cond − uncond) logits and top-k sampling, then decodes the codes with EnCodec.

The T5 encoder and EnCodec are implemented natively; their published weights load with LoadTextEncoderWeights and the codec’s own loader, and the language model’s with LoadLanguageModelWeights (audiocraft state-dict names).

Example

var musicGen = new MusicGenModel<float>(architecture, new MusicGenOptions { ModelSize = MusicGenModelSize.Small });
musicGen.Train(new MusicGenTrainingSample<float> { Audio = clip, Description = "lofi hip hop with a mellow piano" });
var music = musicGen.GenerateMusic("an upbeat 80s synth-pop track", durationSeconds: 10);

Constructors

Constructor Summary
MusicGenModel(NeuralNetworkArchitecture<T> architecture, MusicGenOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable MusicGen.

Properties

Property Summary
ValueTuple<int, int, int> Dimensions { get; } The width D, heads and layers of the configured size (preset unless overridden).
bool IsOnnxMode { get; }
bool SupportsAudioContinuation { get; }
bool SupportsAudioInpainting { get; }
bool SupportsTextToAudio { get; }
bool SupportsTextToMusic { get; }
double FrameRate { get; } The codec’s frames per second (50).
double MaxDurationSeconds { get; }
int SampleRate { get; }
string PaperOptimizerVariant { get; }

Methods

Method Summary
AudioGenerationOptions<T> GetDefaultOptions()
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(Tensor<T> input, Tensor<T> target) The training objective on codes and token ids, deterministic and without dropout.
T Train(MusicGenTrainingSample<T> sample) Trains one step on a clip and its description; returns the loss.
Task<Tensor<T>> GenerateAudioAsync(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed, CancellationToken cancellationToken)
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed)
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed)
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed)
Tensor<T> GenerateWithMelody(string prompt, Tensor<T> melody, double durationSeconds, double guidanceScale, int? seed) Generates music following a melody and a description (the melody model, §2.3): the melody’s chromagram, quantized to its dominant pitch class per frame, conditions the generation.
Tensor<T> InpaintAudio(Tensor<T> audio, Tensor<T> mask, string prompt, int numInferenceSteps, int? seed)
Tensor<T> TextToTokens(string text) The T5 token ids of text (t5-base’s tokenizer, ending with </s>).
int[] GenerateCodes(Tensor<T> tokens, Tensor<T> melody, int frames, int seed, double? guidanceScale, int[] prompt) Generates codes conditioned on token ids and, for a melody model, a melody waveform.
int[] GenerateCodes(Tensor<T> tokens, int frames, int seed) Calls the fully specified overload with the omitted parameters’ defaults.
int[] GenerateCodes(Tensor<T> tokens, int frames, int seed, double? guidanceScale) Calls the fully specified overload with the omitted parameters’ defaults.
int[] GenerateCodes(Tensor<T> tokens, int frames, int seed, double? guidanceScale, int[] prompt) Generates codes [codebooks, frames] for token ids (empty for unconditional generation), continuing an optional prompt of codes [codebooks, prompt frames].
int[] GenerateCodes(Tensor<T> tokens, int frames, int seed, int[] prompt) Calls the fully specified overload with the omitted parameters’ defaults.
void LoadLanguageModelWeights(Func<string, int[], double[]> read) Loads an audiocraft MusicGen language model straight from source.
void LoadTextEncoderWeights(Func<string, int[], double[]> read) Loads a Hugging Face T5 encoder state dict (T5EncoderModel, e.g.