MedicalSAM2<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Medical
Medical SAM 2: SAM 2 for medical images and volumes, segmenting sets of 2D images as pseudo-videos through a self-sorting memory bank.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → MedicalSegmentationBase<T> → MedicalSAM2<T>
For Beginners
Prompt one image of a set (for example, click on the optic disc in one retina photo) and Medical SAM 2 segments the same structure in all the others, using the most varied, confident earlier results as its memory. For a 3D scan, prompt slices and it tracks the structure through the volume.
How It Works
Medical SAM 2 (Zhu et al. 2024) is SAM 2 (the official code’s Hiera-S, at the paper’s 256 pixels) on the shared network
verified against the official SAM 2 code. For a set of images it replaces SAM 2’s recent-frame memory with a
self-sorting bank: a confident prediction’s memory joins the bank’s candidates, the K most mutually dissimilar are kept
(eqs. 2-3), and each image attends to K memories drawn by similarity (eqs. 4-5). A volume is tracked as videos along
each prompted axis, in both orders, and the orientations’ probabilities averaged. Training is SAM 2’s (see
MedicalSAM2Options), with the bank as the memory for image sets.
Reference: Zhu et al., “Medical SAM 2: Segment medical images as video via Segment Anything Model 2”, arXiv 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.BinaryClassification,
inputHeight: 256, inputWidth: 256, inputDepth: 3, outputSize: 1);
var model = new MedicalSAM2<float>(architecture);
var masks = model.SegmentImageSet(images, templateIndex: 0, points: new float[,] { { 120, 96 } }, labels: new[] { 1 });
Constructors
| Constructor | Summary |
|---|---|
MedicalSAM2(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, MedicalSAM2Options options) |
Creates a trainable Medical SAM 2. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
List<Tensor<T>> SegmentImageSet(List<Tensor<T>> images, int templateIndex, Tensor<T> mask) |
Segments a structure through a set of images from its mask [H, W] on one of them. |
List<Tensor<T>> SegmentImageSet(List<Tensor<T>> images, int templateIndex, float[] box) |
Segments a structure through a set of images from its mask [H, W] on one of them. |
List<Tensor<T>> SegmentImageSet(List<Tensor<T>> images, int templateIndex, float[] points, int[] labels) |
Segments a structure through a set of images [C, H, W] (values in 0..1) from clicks on one of them (the one-prompt setting): the images are processed in order as a pseudo-video, each conditioned on the self-sorting bank. |
MedicalSegmentationResult<T> SegmentSlice(Tensor<T> slice) |
An image [C, H, W] without a prompt or memory: the decoder on the model’s empty prompt. |
MedicalSegmentationResult<T> SegmentVolume(Tensor<T> volume) |
A volume [C, D, H, W] (or a slice) without a prompt: the axial slices tracked in both orders from the model’s empty prompt on the first slice of each, and averaged. |
MedicalSegmentationResult<T> SegmentVolume(Tensor<T> volume, IReadOnlyDictionary<int, Tensor<T>> axialMasks, IReadOnlyDictionary<int, Tensor<T>> coronalMasks, IReadOnlyDictionary<int, Tensor<T>> sagittalMasks) |
Segments a structure through a volume [C, D, H, W] (values in 0..1) from its mask on some slices along each axis: axial slices [H, W] keyed by depth, and optionally coronal slices [D, W] keyed by row and sagittal slices [D, H] keyed… |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
void LoadPretrainedWeights(string path) |
Loads a checkpoint in Hugging Face’s Sam2VideoModel format (an official SAM 2 or Medical SAM 2 .pt converted with transformers’ SAM 2 conversion script). |
void SetTrainingMode(bool isTraining) |
|
void TrainOnVolume(Tensor<T> volume, Tensor<T> masks) |
One step of SAM 2’s video training on a structure through consecutive axial slices of a volume [C, D, H, W] and its masks [D, H, W] (the paper samples 8-slice sequences). |