AudioFlamingo2<T>
Models & Types · AiDotNet.Audio.Multimodal
Audio Flamingo 2 multimodal audio-language model for audio understanding with interleaved inputs.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioFlamingo2<T>
Implements: IAudioLanguageModel<T>
For Beginners
Audio Flamingo 2 gives a language AI the ability to hear. It can listen to audio recordings and answer questions about them, generate descriptions, or reason about what’s happening in the audio scene. It works by connecting a pre-trained audio encoder to a language model using a special adapter layer.
Usage:
How It Works
Audio Flamingo 2 (2024) extends the Flamingo architecture for audio understanding with interleaved audio-text inputs. It uses a frozen audio encoder with perceiver-style cross-attention to adapt a pre-trained LLM for audio captioning, QA, and reasoning.
Constructors
| Constructor | Summary |
|---|---|
AudioFlamingo2(NeuralNetworkArchitecture<T> architecture, AudioFlamingo2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an Audio Flamingo 2 model in native training mode. |
AudioFlamingo2(NeuralNetworkArchitecture<T> architecture, string modelPath, AudioFlamingo2Options options) |
Creates an Audio Flamingo 2 model in native training mode. |
Properties
| Property | Summary |
|---|---|
double MaxAudioDurationSeconds { get; } |
|
int MaxResponseTokens { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<string> GetCapabilities() |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<string> UnderstandAsync(Tensor<T> audio, string prompt, int maxTokens, double temperature, CancellationToken cancellationToken) |
|
Tensor<T> ExtractAudioEmbeddings(Tensor<T> audio) |
|
string Caption(Tensor<T> audio, int maxTokens) |
|
string Understand(Tensor<T> audio, string prompt, int maxTokens, double temperature) |