AudioFlamingo2<T>

Models & Types · AiDotNet.Audio.Multimodal

Audio Flamingo 2 multimodal audio-language model for audio understanding with interleaved inputs.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioFlamingo2<T>

Implements: IAudioLanguageModel<T>

For Beginners

Audio Flamingo 2 gives a language AI the ability to hear. It can listen to audio recordings and answer questions about them, generate descriptions, or reason about what’s happening in the audio scene. It works by connecting a pre-trained audio encoder to a language model using a special adapter layer.

Usage:

How It Works

Audio Flamingo 2 (2024) extends the Flamingo architecture for audio understanding with interleaved audio-text inputs. It uses a frozen audio encoder with perceiver-style cross-attention to adapt a pre-trained LLM for audio captioning, QA, and reasoning.

Constructors

Constructor Summary
AudioFlamingo2(NeuralNetworkArchitecture<T> architecture, AudioFlamingo2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an Audio Flamingo 2 model in native training mode.
AudioFlamingo2(NeuralNetworkArchitecture<T> architecture, string modelPath, AudioFlamingo2Options options) Creates an Audio Flamingo 2 model in native training mode.

Properties

Property Summary
double MaxAudioDurationSeconds { get; }
int MaxResponseTokens { get; }

Methods

Method Summary
IReadOnlyList<string> GetCapabilities()
ModelMetadata<T> GetModelMetadata()
Task<string> UnderstandAsync(Tensor<T> audio, string prompt, int maxTokens, double temperature, CancellationToken cancellationToken)
Tensor<T> ExtractAudioEmbeddings(Tensor<T> audio)
string Caption(Tensor<T> audio, int maxTokens)
string Understand(Tensor<T> audio, string prompt, int maxTokens, double temperature)