BridgeTower<T>

Models & Types · AiDotNet.VisionLanguage.Foundational

BridgeTower: cross-modal alignment through bridge layers connecting vision and text encoder layers.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → BridgeTower<T>

Implements: IVisionLanguageFusionModel<T>

For Beginners

BridgeTower is a vision-language model. Default values follow the original paper settings.

How It Works

BridgeTower (Xu et al., AAAI 2023) introduces bridge layers that connect vision and text encoder layers at multiple levels, enabling fine-grained cross-modal alignment. Each bridge layer consists of cross-attention between corresponding encoder layers, creating bidirectional information flow throughout the encoding process.

References:

  • Paper: “BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning” (Xu et al., AAAI 2023)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new BridgeTower<double>(architecture, new BridgeTowerOptions());

Constructors

Constructor Summary
BridgeTower(NeuralNetworkArchitecture<T> architecture, BridgeTowerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
BridgeTower(NeuralNetworkArchitecture<T> architecture, string modelPath, BridgeTowerOptions options)

Properties

Property Summary
int EmbeddingDimension { get; }
int FusionEmbeddingDim { get; }
int MaxSequenceLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T ComputeMatchingScore(Tensor<T> image, string text)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> FuseImageText(Tensor<T> image, string text)