Concerto<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.PointCloud
Concerto: Hybrid Mamba-Transformer backbone for 3D point clouds.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → SemanticSegmentationBase<T> → Concerto<T>
For Beginners
High-accuracy 3D point cloud segmentation. Multi-modal 3D scene understanding.
Common use cases:
- High-accuracy 3D point cloud segmentation
- Multi-modal 3D scene understanding
- Dense 3D prediction tasks
- Advanced LiDAR perception
How It Works
Technical Details:
- Hybrid Mamba + Transformer blocks for balanced efficiency and accuracy
- Global Mamba branches + local Transformer attention
- Better quality than pure Mamba while remaining efficient
- Serialized point processing with space-filling curves
Reference: “Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations” (arXiv:2510.23607).
Migration in progress — this class does not yet implement that paper. Its attribution
was previously wrong in three separate ways: a fabricated reference (“Sonata and Concerto:
Mamba for 3D Point Clouds”, which does not exist), a [ResearchPaper] attribute pointing
at Mamba3D (arXiv:2404.14966 — a real but unrelated model by different authors), and an
implementation matching neither. The citation now names the real Concerto, and
ConcertoOptions carries that paper’s hyperparameters.
What the published method actually is, and what still has to be built here:
Because Concerto is a PRETRAINING method, segmentation is a downstream probe on the learned
representation rather than the training objective. ISemanticSegmentation is
retained so existing consumers keep working, but the current backbone is the previous
unattributed hybrid Mamba/Transformer stack, not PTv3, and no self-supervised objective runs.
Example
var architecture = new NeuralNetworkArchitecture<float>(
InputType.ThreeDimensional, NeuralNetworkTaskType.ImageSegmentation,
inputSize: 6, outputSize: 40);
var trainPoints = Tensor<float>.CreateRandom(4, 128, 6);
var trainLabels = Tensor<float>.CreateRandom(4, 128, 40);
// Every tunable is set through the options object; these are the defaults.
var options = new ConcertoOptions { NumClasses = 40, DropRate = 0.1, ModelSize = ConcertoModelSize.Base };
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new Concerto<float>(architecture, options: options))
.Build(trainPoints, trainLabels);
var pointCloudTensor = Tensor<float>.CreateRandom(1, 128, 6);
Tensor<float> segmentationMap = result.Predict(pointCloudTensor);
Constructors
| Constructor | Summary |
|---|---|
Concerto(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, ConcertoOptions options) |
Initializes Concerto in native (trainable) mode. |
Concerto(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, ConcertoOptions options) |
Initializes Concerto in ONNX (inference-only) mode. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
Collects metadata describing this model’s configuration. |
T Pretrain(IReadOnlyList<ConcertoPretrainingSample<T>> samples, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Runs Concerto’s self-supervised pretraining: cross-modal distillation from paired camera views plus intra-modal self-distillation against a momentum teacher. |