Pengi<T>
Models & Types · AiDotNet.Audio.Multimodal
Pengi audio language model that frames all audio tasks as text-generation.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Pengi<T>
Implements: IAudioLanguageModel<T>
For Beginners
Pengi treats all audio understanding as a conversation. Instead of having separate models for “what sound is this?” and “describe this audio”, Pengi uses one model that can answer any question about audio by generating text responses. Think of it as a chat assistant that can hear audio.
Usage:
How It Works
Pengi (Deshmukh et al., 2023, Microsoft) is an audio language model that frames all audio tasks as text-generation tasks. It uses a frozen Audio Spectrogram Transformer (AST) encoder paired with a pre-trained language model, enabling open-ended audio reasoning, captioning, and question answering without task-specific classification heads.
Constructors
| Constructor | Summary |
|---|---|
Pengi(NeuralNetworkArchitecture<T> architecture, PengiOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a Pengi model in native training mode. |
Pengi(NeuralNetworkArchitecture<T> architecture, string modelPath, PengiOptions options) |
Creates a Pengi model in native training mode. |
Properties
| Property | Summary |
|---|---|
double MaxAudioDurationSeconds { get; } |
|
int MaxResponseTokens { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<string> GetCapabilities() |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<string> UnderstandAsync(Tensor<T> audio, string prompt, int maxTokens, double temperature, CancellationToken cancellationToken) |
|
Tensor<T> ExtractAudioEmbeddings(Tensor<T> audio) |
|
string Caption(Tensor<T> audio, int maxTokens) |
|
string Understand(Tensor<T> audio, string prompt, int maxTokens, double temperature) |