Shikra<T>
Models & Types · AiDotNet.VisionLanguage.Grounding
Shikra: referential dialogue model with coordinate output for grounding.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Shikra<T>
Implements: IGroundingTrainable<T>, IVisualGroundingModel<T>
For Beginners
Shikra is a vision-language model that grounds objects by generating their coordinates as plain text numbers in natural dialogue. Default values follow the original paper settings.
How It Works
Shikra (Chen et al., 2023) is a referential dialogue model that treats spatial coordinates as plain-text numbers in the LLM output vocabulary. Instead of using special tokens or external detection heads, it generates normalized bounding box coordinates as decimal text tokens via CLIP ViT visual encoding interleaved with instruction tokens, enabling natural language grounding through straightforward autoregressive generation.
References:
- Paper: “Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic” (SenseTime, 2023)
Example
// Create a Shikra model for referential dialogue with coordinate grounding
// generating bounding box coordinates as plain-text numbers
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new Shikra<double>(architecture, "shikra.onnx");
// Training mode with native layers
var trainModel = new Shikra<double>(architecture, new ShikraOptions());
Constructors
| Constructor | Summary |
|---|---|
Shikra(NeuralNetworkArchitecture<T> architecture, ShikraOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Shikra(NeuralNetworkArchitecture<T> architecture, string modelPath, ShikraOptions options) |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
|
int MaxDetections { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateGroundingLoss(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) |
The loss of TrainGrounding on the same supervision, without training. |
Tensor<T> DetectObjects(Tensor<T> image, IReadOnlyList<string> categories) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GroundText(Tensor<T> image, string textQuery) |
Grounds text using Shikra’s referential dialogue with numeric coordinate output. |
void TrainGrounding(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) |
One training step of the grounding head and the language model with Shikra (Chen et al. |