ConcertoCrossModalObjective<T>

Helpers & Utilities · AiDotNet.ComputerVision.Segmentation.PointCloud

Concerto’s 2D-3D cross-modal joint-embedding objective (arXiv:2510.23607).

How It Works

Aligns 3D point features with frozen 2D image-patch features. Correspondence is established exactly as the paper describes it — a 3D-to-2D projection followed by depth-based visibility verification — after which the point features falling inside each patch are mean-pooled and matched to that patch’s image embedding by cosine similarity.

Two details from the paper are load-bearing and easy to get wrong:

Methods

Method Summary
T ComputeLoss(Tensor<T> pointFeatures, Tensor<T> pointCoordinates, IReadOnlyList<ConcertoPairedView<T>> views, double depthToleranceMeters) Computes the mean cosine-similarity loss between point-derived patch features and the frozen image encoder’s patch features, over every supplied view.
Tensor<T> ComputeTapeLoss(Tensor<T> pointFeatures, Tensor<T> pointCoordinates, IReadOnlyList<ConcertoPairedView<T>> views, double depthToleranceMeters) The same objective as ComputeLoss, expressed in engine operations so a GradientTape can differentiate it with respect to the point features.