RTDETR<T>

Models & Types · AiDotNet.ComputerVision.Detection.ObjectDetection.DETR

RT-DETR (Real-Time DEtection TRansformer) - the first real-time end-to-end object detector.

Inheritance: ModelBase<T, Tensor<T>, Tensor<T>> → VisionTaskModelBase<T> → ObjectDetectorBase<T> → RTDETR<T>

Implements: IDetectionTrainingModel<T>

For Beginners

RT-DETR predicts a fixed set of boxes end to end, like DETR, but at YOLO speed. It gets there two ways. A hybrid encoder runs full attention only on the coarsest feature map and fuses the scales with convolutions. The decoder starts from the encoder’s highest-scoring positions instead of learned queries. No NMS is needed.

How It Works

The architecture follows the reference implementation (lyuwenyu/RT-DETR, rtdetr_pytorch):

  • ResNet C3-C5.
  • The hybrid encoder: a ConvNorm projection, AIFI on S5 with a 2-D sin-cos encoding, and a CCFM

FPN + PAN of CSPRep blocks.

  • IoU-aware query selection of the top 300 encoder tokens, whose detached features become the

content queries.

  • Decoder layers with MultiScaleDeformableAttention (3 levels x 4 points) and

per-layer heads with iterative refinement.

  • Contrastive denoising.

The training objective is the reference’s: varifocal + L1 + GIoU with Hungarian matching on every decoder layer and on the encoder’s selected queries, plus the fixed-assignment denoising loss of every layer.

Reference: Zhao et al., “DETRs Beat YOLOs on Real-time Object Detection”, CVPR 2024

Constructors

Constructor Summary
RTDETR(ObjectDetectionOptions<T> options) Creates RT-DETR.

Properties

Property Summary
string Name { get; }

Methods

Method Summary
DetectionResult<T> Detect(Tensor<T> image, double confidenceThreshold, double nmsThreshold)
Task LoadWeightsAsync(string pathOrUrl, CancellationToken cancellationToken)
double EffectiveNmsThreshold(double requested)
void SaveWeights(string path)
void SetTrainingMode(bool training)
void Train(Tensor<T> input, Tensor<T> expectedOutput)
void TrainDetections(Tensor<T> input, DetectionTrainingBatch<T> targets)