MusicFlamingo<T>

Models & Types · AiDotNet.Audio.Multimodal

Music Flamingo multimodal music-language model for music understanding and reasoning.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → MusicFlamingo<T>

Implements: IAudioLanguageModel<T>

For Beginners

Music Flamingo gives a language AI the ability to understand music. You can play it a song and ask “What genre is this?” or “What instruments are playing?” and it answers in natural language by combining its music listening with language understanding.

Usage:

How It Works

Music Flamingo (2024) adapts the Flamingo architecture specifically for music understanding. It uses a frozen music encoder (e.g., MERT or Jukebox features) with perceiver cross-attention to enable a pre-trained LLM to reason about music: answering questions about genre, instruments, mood, structure, and musical theory.

Constructors

Constructor Summary
MusicFlamingo(NeuralNetworkArchitecture<T> architecture, MusicFlamingoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a Music Flamingo model in native training mode.
MusicFlamingo(NeuralNetworkArchitecture<T> architecture, string modelPath, MusicFlamingoOptions options) Creates a Music Flamingo model in native training mode.

Properties

Property Summary
double MaxAudioDurationSeconds { get; }
int MaxResponseTokens { get; }

Methods

Method Summary
IReadOnlyList<string> GetCapabilities()
ModelMetadata<T> GetModelMetadata()
Task<string> UnderstandAsync(Tensor<T> audio, string prompt, int maxTokens, double temperature, CancellationToken cancellationToken)
Tensor<T> ExtractAudioEmbeddings(Tensor<T> audio)
string Caption(Tensor<T> audio, int maxTokens)
string Understand(Tensor<T> audio, string prompt, int maxTokens, double temperature)