Concerto<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.PointCloud

Concerto: Hybrid Mamba-Transformer backbone for 3D point clouds.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → SemanticSegmentationBase<T> → Concerto<T>

For Beginners

High-accuracy 3D point cloud segmentation. Multi-modal 3D scene understanding.

Common use cases:

  • High-accuracy 3D point cloud segmentation
  • Multi-modal 3D scene understanding
  • Dense 3D prediction tasks
  • Advanced LiDAR perception

How It Works

Technical Details:

  • Hybrid Mamba + Transformer blocks for balanced efficiency and accuracy
  • Global Mamba branches + local Transformer attention
  • Better quality than pure Mamba while remaining efficient
  • Serialized point processing with space-filling curves

Reference: “Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations” (arXiv:2510.23607).

Migration in progress — this class does not yet implement that paper. Its attribution was previously wrong in three separate ways: a fabricated reference (“Sonata and Concerto: Mamba for 3D Point Clouds”, which does not exist), a [ResearchPaper] attribute pointing at Mamba3D (arXiv:2404.14966 — a real but unrelated model by different authors), and an implementation matching neither. The citation now names the real Concerto, and ConcertoOptions carries that paper’s hyperparameters.

What the published method actually is, and what still has to be built here:

Because Concerto is a PRETRAINING method, segmentation is a downstream probe on the learned representation rather than the training objective. ISemanticSegmentation is retained so existing consumers keep working, but the current backbone is the previous unattributed hybrid Mamba/Transformer stack, not PTv3, and no self-supervised objective runs.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    InputType.ThreeDimensional, NeuralNetworkTaskType.ImageSegmentation,
    inputSize: 6, outputSize: 40);
            
var trainPoints = Tensor<float>.CreateRandom(4, 128, 6);
var trainLabels = Tensor<float>.CreateRandom(4, 128, 40);
            
// Every tunable is set through the options object; these are the defaults.
var options = new ConcertoOptions { NumClasses = 40, DropRate = 0.1, ModelSize = ConcertoModelSize.Base };
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new Concerto<float>(architecture, options: options))
    .Build(trainPoints, trainLabels);
            
var pointCloudTensor = Tensor<float>.CreateRandom(1, 128, 6);
Tensor<float> segmentationMap = result.Predict(pointCloudTensor);

Constructors

Constructor Summary
Concerto(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, ConcertoOptions options) Initializes Concerto in native (trainable) mode.
Concerto(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ConcertoOptions options) Initializes Concerto in ONNX (inference-only) mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Collects metadata describing this model’s configuration.
T Pretrain(IReadOnlyList<ConcertoPretrainingSample<T>> samples, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Runs Concerto’s self-supervised pretraining: cross-modal distillation from paired camera views plus intra-modal self-distillation against a momentum teacher.