MusicFlamingo<T>
Models & Types · AiDotNet.Audio.Multimodal
Music Flamingo multimodal music-language model for music understanding and reasoning.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → MusicFlamingo<T>
Implements: IAudioLanguageModel<T>
For Beginners
Music Flamingo gives a language AI the ability to understand music. You can play it a song and ask “What genre is this?” or “What instruments are playing?” and it answers in natural language by combining its music listening with language understanding.
Usage:
How It Works
Music Flamingo (2024) adapts the Flamingo architecture specifically for music understanding. It uses a frozen music encoder (e.g., MERT or Jukebox features) with perceiver cross-attention to enable a pre-trained LLM to reason about music: answering questions about genre, instruments, mood, structure, and musical theory.
Constructors
| Constructor | Summary |
|---|---|
MusicFlamingo(NeuralNetworkArchitecture<T> architecture, MusicFlamingoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a Music Flamingo model in native training mode. |
MusicFlamingo(NeuralNetworkArchitecture<T> architecture, string modelPath, MusicFlamingoOptions options) |
Creates a Music Flamingo model in native training mode. |
Properties
| Property | Summary |
|---|---|
double MaxAudioDurationSeconds { get; } |
|
int MaxResponseTokens { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<string> GetCapabilities() |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<string> UnderstandAsync(Tensor<T> audio, string prompt, int maxTokens, double temperature, CancellationToken cancellationToken) |
|
Tensor<T> ExtractAudioEmbeddings(Tensor<T> audio) |
|
string Caption(Tensor<T> audio, int maxTokens) |
|
string Understand(Tensor<T> audio, string prompt, int maxTokens, double temperature) |