Pengi<T>

Models & Types · AiDotNet.Audio.Multimodal

Pengi audio language model that frames all audio tasks as text-generation.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Pengi<T>

Implements: IAudioLanguageModel<T>

For Beginners

Pengi treats all audio understanding as a conversation. Instead of having separate models for “what sound is this?” and “describe this audio”, Pengi uses one model that can answer any question about audio by generating text responses. Think of it as a chat assistant that can hear audio.

Usage:

How It Works

Pengi (Deshmukh et al., 2023, Microsoft) is an audio language model that frames all audio tasks as text-generation tasks. It uses a frozen Audio Spectrogram Transformer (AST) encoder paired with a pre-trained language model, enabling open-ended audio reasoning, captioning, and question answering without task-specific classification heads.

Constructors

Constructor Summary
Pengi(NeuralNetworkArchitecture<T> architecture, PengiOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a Pengi model in native training mode.
Pengi(NeuralNetworkArchitecture<T> architecture, string modelPath, PengiOptions options) Creates a Pengi model in native training mode.

Properties

Property Summary
double MaxAudioDurationSeconds { get; }
int MaxResponseTokens { get; }

Methods

Method Summary
IReadOnlyList<string> GetCapabilities()
ModelMetadata<T> GetModelMetadata()
Task<string> UnderstandAsync(Tensor<T> audio, string prompt, int maxTokens, double temperature, CancellationToken cancellationToken)
Tensor<T> ExtractAudioEmbeddings(Tensor<T> audio)
string Caption(Tensor<T> audio, int maxTokens)
string Understand(Tensor<T> audio, string prompt, int maxTokens, double temperature)