AudioTextDualStreamArchitecture<T>

Models & Types · AiDotNet.NeuralNetworks

A neural network architecture for audio + text two-stream models (CLAP-family encoders) that hosts a separate audio encoder and text encoder.

Inheritance: NeuralNetworkArchitecture<T> → DualEncoderArchitecture<T> → AudioTextDualStreamArchitecture<T>

How It Works

Concrete audio-language specialisation of DualEncoderArchitecture. The abstract base owns the two encoder stacks under modality-neutral names (EncoderALayers / EncoderBLayers); this subclass exposes them under their semantic aliases AudioLayers / TextLayers so CLAP-family code reads naturally.

CLAP (Wu et al. 2023) and similar audio-language pretraining models train two parallel encoders: an audio side (HTSAT / PANNs / Wav2Vec / etc.) and a text side (RoBERTa / BERT / etc.). They only meet at the contrastive objective, so each side gets its own customisable layer stack.

Constructors

Constructor Summary
AudioTextDualStreamArchitecture(IEnumerable<ILayer<T>> audioLayers, IEnumerable<ILayer<T>> textLayers, InputType inputType, NeuralNetworkTaskType taskType, NetworkComplexity complexity, int inputSize, int outputSize, int audioEmbeddingDim, int textEmbeddingDim, bool shouldReturnFullSequence) Initializes a new audio-text dual-stream architecture with explicit audio and text encoder layer stacks.

Properties

Property Summary
IReadOnlyList<ILayer<T>> AudioLayers { get; } Gets the layer stack for the audio encoder.
IReadOnlyList<ILayer<T>> TextLayers { get; } Gets the layer stack for the text encoder.