Ferret<T>
Models & Types · AiDotNet.VisionLanguage.Grounding
Ferret: spatial-aware visual sampler for free-form region referring and grounding.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Ferret<T>
Implements: IGroundingTrainable<T>, IVisualGroundingModel<T>
For Beginners
Ferret is a vision-language model that can locate objects in images based on natural language descriptions and refer to regions of any shape. Default values follow the original paper settings.
How It Works
Ferret (You et al., 2023) introduces a spatial-aware visual sampler that can refer to and ground anything at any granularity. It uses a hybrid region representation combining discrete coordinates with continuous visual features and a spatial-aware visual sampler to handle free-form referring regions of any shape including points, boxes, and scribbles.
References:
- Paper: “Ferret: Refer and Ground Anything Anywhere at Any Granularity” (Apple, 2023)
Example
// Create a Ferret model for region referring and grounding
// with spatial-aware visual sampler at any granularity
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new Ferret<double>(architecture, "ferret.onnx");
// Training mode with native layers
var trainModel = new Ferret<double>(architecture, new FerretOptions());
Constructors
| Constructor | Summary |
|---|---|
Ferret(NeuralNetworkArchitecture<T> architecture, FerretOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Ferret(NeuralNetworkArchitecture<T> architecture, string modelPath, FerretOptions options) |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
|
int MaxDetections { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateGroundingLoss(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) |
The loss of TrainGrounding on the same supervision, without training. |
Tensor<T> DetectObjects(Tensor<T> image, IReadOnlyList<string> categories) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GroundText(Tensor<T> image, string textQuery) |
Grounds text using Ferret’s spatial-aware visual sampler for free-form referring. |
void TrainGrounding(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) |
One training step of the grounding head and the language model with Ferret (You et al. |