AudioTextDualStreamArchitecture<T>
Models & Types · AiDotNet.NeuralNetworks
A neural network architecture for audio + text two-stream models (CLAP-family encoders) that hosts a separate audio encoder and text encoder.
Inheritance: NeuralNetworkArchitecture<T> → DualEncoderArchitecture<T> → AudioTextDualStreamArchitecture<T>
How It Works
Concrete audio-language specialisation of DualEncoderArchitecture.
The abstract base owns the two encoder stacks under modality-neutral names
(EncoderALayers / EncoderBLayers); this subclass exposes them
under their semantic aliases AudioLayers / TextLayers so
CLAP-family code reads naturally.
CLAP (Wu et al. 2023) and similar audio-language pretraining models train two parallel encoders: an audio side (HTSAT / PANNs / Wav2Vec / etc.) and a text side (RoBERTa / BERT / etc.). They only meet at the contrastive objective, so each side gets its own customisable layer stack.
Constructors
| Constructor | Summary |
|---|---|
AudioTextDualStreamArchitecture(IEnumerable<ILayer<T>> audioLayers, IEnumerable<ILayer<T>> textLayers, InputType inputType, NeuralNetworkTaskType taskType, NetworkComplexity complexity, int inputSize, int outputSize, int audioEmbeddingDim, int textEmbeddingDim, bool shouldReturnFullSequence) |
Initializes a new audio-text dual-stream architecture with explicit audio and text encoder layer stacks. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<ILayer<T>> AudioLayers { get; } |
Gets the layer stack for the audio encoder. |
IReadOnlyList<ILayer<T>> TextLayers { get; } |
Gets the layer stack for the text encoder. |