Qwen2Audio<T>

Models & Types · AiDotNet.Audio.Multimodal

Qwen2-Audio multimodal audio-language model for audio understanding and reasoning.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Qwen2Audio<T>

Implements: IAudioLanguageModel<T>

For Beginners

Qwen2-Audio can “listen” to audio and answer questions about it. Play it music and ask “What genre is this?”, play it a conversation and ask “What language are they speaking?”, or play environmental sounds and ask “Describe this scene.”

Usage:

How It Works

Qwen2-Audio (Chu et al., 2024, Alibaba) uses a Whisper-style audio encoder with a Qwen2 language model backbone, connected by a perceiver-style adapter. It supports audio captioning, question answering, sound event detection, and audio reasoning.

Constructors

Constructor Summary
Qwen2Audio(NeuralNetworkArchitecture<T> architecture, Qwen2AudioOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a Qwen2-Audio model in native training mode.
Qwen2Audio(NeuralNetworkArchitecture<T> architecture, string modelPath, Qwen2AudioOptions options) Creates a Qwen2-Audio model in native training mode.

Properties

Property Summary
double MaxAudioDurationSeconds { get; }
int MaxResponseTokens { get; }

Methods

Method Summary
IReadOnlyList<string> GetCapabilities()
ModelMetadata<T> GetModelMetadata()
Task<string> UnderstandAsync(Tensor<T> audio, string prompt, int maxTokens, double temperature, CancellationToken cancellationToken)
Tensor<T> ExtractAudioEmbeddings(Tensor<T> audio)
string Caption(Tensor<T> audio, int maxTokens)
string Understand(Tensor<T> audio, string prompt, int maxTokens, double temperature)