MusicGenModel<T>
Models & Types · AiDotNet.Audio.MusicGen
MusicGen (Copet et al. 2023): text-to-music generation with one transformer language model over the four codebooks of a 32 kHz EnCodec, interleaved by the “delay” pattern and conditioned on frozen T5 text embeddings through cross-attention, sampled with classifier-free guidance.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → MusicGenModel<T>
Implements: IAudioGenerator<T>, IPaperOptimizerVariant
For Beginners
Describe the music you want and MusicGen writes it: it predicts compressed audio a frame at a time, steered toward the description, and a codec turns those frames into sound.
How It Works
The text is tokenized with t5-base’s SentencePiece model and encoded by a T5 encoder (frozen, as the paper’s
pretrained conditioner), projected to the model width. The language model (MusicGenCore) reads the
delay-interleaved codes and is trained with each codebook’s cross-entropy over its valid steps. Training applies the
paper’s condition merging, description dropout, word dropout and condition dropout; the optimizer is AdamW (or
D-Adaptation for the 300M model) with clipping and a cosine schedule, and an EMA of the weights is what generation
uses once training has updated it. Generation fills the pattern step by step with
uncond + 3 · (cond − uncond) logits and top-k sampling, then decodes the codes with EnCodec.
The T5 encoder and EnCodec are implemented natively; their published weights load with
LoadTextEncoderWeights and the codec’s own loader, and the language model’s with
LoadLanguageModelWeights (audiocraft state-dict names).
Example
var musicGen = new MusicGenModel<float>(architecture, new MusicGenOptions { ModelSize = MusicGenModelSize.Small });
musicGen.Train(new MusicGenTrainingSample<float> { Audio = clip, Description = "lofi hip hop with a mellow piano" });
var music = musicGen.GenerateMusic("an upbeat 80s synth-pop track", durationSeconds: 10);
Constructors
| Constructor | Summary |
|---|---|
MusicGenModel(NeuralNetworkArchitecture<T> architecture, MusicGenOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable MusicGen. |
Properties
| Property | Summary |
|---|---|
ValueTuple<int, int, int> Dimensions { get; } |
The width D, heads and layers of the configured size (preset unless overridden). |
bool IsOnnxMode { get; } |
|
bool SupportsAudioContinuation { get; } |
|
bool SupportsAudioInpainting { get; } |
|
bool SupportsTextToAudio { get; } |
|
bool SupportsTextToMusic { get; } |
|
double FrameRate { get; } |
The codec’s frames per second (50). |
double MaxDurationSeconds { get; } |
|
int SampleRate { get; } |
|
string PaperOptimizerVariant { get; } |
Methods
| Method | Summary |
|---|---|
AudioGenerationOptions<T> GetDefaultOptions() |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(Tensor<T> input, Tensor<T> target) |
The training objective on codes and token ids, deterministic and without dropout. |
T Train(MusicGenTrainingSample<T> sample) |
Trains one step on a clip and its description; returns the loss. |
Task<Tensor<T>> GenerateAudioAsync(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed, CancellationToken cancellationToken) |
|
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed) |
|
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
|
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
|
Tensor<T> GenerateWithMelody(string prompt, Tensor<T> melody, double durationSeconds, double guidanceScale, int? seed) |
Generates music following a melody and a description (the melody model, §2.3): the melody’s chromagram, quantized to its dominant pitch class per frame, conditions the generation. |
Tensor<T> InpaintAudio(Tensor<T> audio, Tensor<T> mask, string prompt, int numInferenceSteps, int? seed) |
|
Tensor<T> TextToTokens(string text) |
The T5 token ids of text (t5-base’s tokenizer, ending with </s>). |
int[] GenerateCodes(Tensor<T> tokens, Tensor<T> melody, int frames, int seed, double? guidanceScale, int[] prompt) |
Generates codes conditioned on token ids and, for a melody model, a melody waveform. |
int[] GenerateCodes(Tensor<T> tokens, int frames, int seed) |
Calls the fully specified overload with the omitted parameters’ defaults. |
int[] GenerateCodes(Tensor<T> tokens, int frames, int seed, double? guidanceScale) |
Calls the fully specified overload with the omitted parameters’ defaults. |
int[] GenerateCodes(Tensor<T> tokens, int frames, int seed, double? guidanceScale, int[] prompt) |
Generates codes [codebooks, frames] for token ids (empty for unconditional generation), continuing an optional prompt of codes [codebooks, prompt frames]. |
int[] GenerateCodes(Tensor<T> tokens, int frames, int seed, int[] prompt) |
Calls the fully specified overload with the omitted parameters’ defaults. |
void LoadLanguageModelWeights(Func<string, int[], double[]> read) |
Loads an audiocraft MusicGen language model straight from source. |
void LoadTextEncoderWeights(Func<string, int[], double[]> read) |
Loads a Hugging Face T5 encoder state dict (T5EncoderModel, e.g. |