MedSAM2<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Medical

MedSAM2: SAM 2.1 fine-tuned to segment anything in 3D medical images and videos.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → MedicalSegmentationBase<T> → MedSAM2<T>

For Beginners

Draw a box around a lesion or organ on one slice of a CT or MRI scan (or one frame of an ultrasound or endoscopy video), and MedSAM2 segments it in that slice and follows it through the rest of the volume.

How It Works

MedSAM2 (Ma et al. 2025, bowang-lab/MedSAM2) is SAM2.1-Tiny at 512 pixels (Hiera blocks 1-2-7-2, four memory-attention layers with 2D RoPE on the 32×32 embedding), fine-tuned on medical volumes and videos. It shares SAM 2’s network, tracking and training (SAM21, verified against the official code at 1e-9). A volume’s slices are frames: grayscale slices are repeated to three channels, and the box on the key slice is propagated forward to the last slice and, starting again from the key slice, backward to the first, as the official inference does. Training follows the paper (AdamW, 3e-5 for the image encoder and 5e-5 for the rest, weight decay 0.01, focal and dice 20:1, boxes moved by up to 10 pixels) and the official config where the paper is silent (see MedSAM2Options). Released checkpoints load with LoadPretrainedWeights once converted to Hugging Face’s format.

Reference: Ma et al., “MedSAM2: Segment Anything in 3D Medical Images and Videos”, arXiv 2025.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.BinaryClassification,
    inputHeight: 512, inputWidth: 512, inputDepth: 1, outputSize: 1);
var model = new MedSAM2<float>(architecture);
model.LoadPretrainedWeights("MedSAM2/model.safetensors");
// volume [1, D, H, W] with values in 0..1 (CT windowed to 0..255, then divided by 255)
var result = model.SegmentVolume(volume, keySlice: 42, box: new float[] { 120, 140, 210, 230 });

Constructors

Constructor Summary
MedSAM2(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, MedSAM2Options options) Creates a trainable MedSAM2.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
MedicalSegmentationResult<T> SegmentSlice(Tensor<T> slice) A slice [C, H, W] without a prompt: the decoder on the model’s empty prompt (a padding point).
MedicalSegmentationResult<T> SegmentSlice(Tensor<T> slice, float[] box) Segments a structure in one slice [C, H, W] from a box [x0, y0, x1, y1] (the image predictor).
MedicalSegmentationResult<T> SegmentVolume(Tensor<T> volume) A volume [C, D, H, W] (or a slice) without a prompt: the middle slice decoded from the model’s empty prompt and propagated both ways.
MedicalSegmentationResult<T> SegmentVolume(Tensor<T> volume, int keySlice, float[] box) Segments a structure through a volume [C, D, H, W] (values in 0..1) from a box [x0, y0, x1, y1] on its key slice: the box is propagated to the last slice and, starting again from the key slice, to the first.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> SegmentVideo(Tensor<T> video, int keyFrame, float[] box) Segments a structure through a video [F, C, H, W] (values in 0..1) from a box on one frame; each frame’s probabilities [F, H, W].
void LoadPretrainedWeights(string path) Loads a checkpoint in Hugging Face’s Sam2VideoModel format (MedSAM2’s released .pt converted with transformers’ SAM 2 conversion script).
void SetTrainingMode(bool isTraining)
void TrainOnVolume(Tensor<T> volume, Tensor<T> masks, bool video) One step of MedSAM2’s training on a structure through consecutive slices of a volume [C, D, H, W] (or frames of a video [F, C, H, W] with video) and its masks [D, H, W]; the paper trains on eight consecutive slices or frames.