TripleStreamArchitecture<T>
Models & Types · AiDotNet.NeuralNetworks
A neural network architecture for three-stream models (e.g., perceiver/abstractor/resampler-style generative VLMs and cross-modality fusion encoders) that hosts a vision encoder, an auxiliary stream (perceiver / abstractor / resampler / cross-modality fusion), and a text or decoder stream.
Inheritance: NeuralNetworkArchitecture<T> → TripleStreamArchitecture<T>
For Beginners
Bigger vision-language models have three pieces — a vision tower, an adapter or “fusion” tower in the middle, and a text or language-decoder tower — that pass data to each other in sequence. If you want to swap in custom layers for all three, this architecture lets you describe each piece independently instead of jamming them into one list (which drops the last two pieces).
How It Works
Models like IDEFICS2, MPLUGOwl3, Qwen3VL, BridgeTower, LXMERT, and METER need three independent
layer stacks. The flat Layers property cannot describe
three graphs, so callers wanting custom layers for all streams should use this type. The
VisionLayers populates the model’s primary Layers list, while AuxiliaryLayers
and TextOrDecoderLayers populate the model-specific auxiliary lists (the perceiver /
abstractor / resampler / cross-modality stack and the language decoder / text encoder
respectively).
Constructors
| Constructor | Summary |
|---|---|
TripleStreamArchitecture(IEnumerable<ILayer<T>> visionLayers, IEnumerable<ILayer<T>> auxiliaryLayers, IEnumerable<ILayer<T>> textOrDecoderLayers, InputType inputType, NeuralNetworkTaskType taskType, NetworkComplexity complexity, int inputSize, int inputHeight, int inputWidth, int inputDepth, int outputSize, bool shouldReturnFullSequence, int imageEmbeddingDim, int textEmbeddingDim, int inputFrames) |
Initializes a new triple-stream architecture with explicit vision, auxiliary, and text/decoder layers. |
Properties
| Property | Summary |
|---|---|
List<ILayer<T>> AuxiliaryLayers { get; } |
Gets the auxiliary stream layer stack (perceiver / abstractor / resampler / cross-modality fusion / bridge depending on model). |
List<ILayer<T>> TextOrDecoderLayers { get; } |
Gets the layer stack for the text encoder or language decoder, depending on model. |
List<ILayer<T>> VisionLayers { get; } |
Gets the layer stack for the vision encoder. |