TextImageAlignmentChecker<T>
Models & Types · AiDotNet.Safety.Multimodal
Checks whether an image matches its caption, to catch unsafe images passed off under a safe description. With a CLIP model it compares CLIP embeddings; without one it falls back to colour heuristics.
Inheritance: ModelBase<T, Vector<T>, Vector<T>> → SafetyModuleBase<T> → TextSafetyModuleBase<T> → TextImageAlignmentChecker<T>
For Beginners
This module checks if what an image shows matches what the text says about it. For example, if someone labels an image “cute puppy” but the image shows something violent, a CLIP model sees that the picture is closer to “graphic violence” than to “cute puppy” and the module reports the mismatch.
How It Works
With a CLIP model (an IMultimodalEmbedding such as ClipNeuralNetwork, which loads
the image and text encoders from ONNX files), EvaluateAlignment runs CLIP zero-shot classification
(Radford et al. 2021): the image embedding is compared with the embeddings of the caption and of three unsafe-concept
prompts (graphic violence, sexually explicit content, hateful symbols), the cosine similarities are scaled by the
CLIP logit scale of 100 and turned into probabilities with a softmax, and a finding is reported when an unsafe
concept takes at least mismatchThreshold of the probability while the caption itself does not describe
unsafe content. The image must already be preprocessed to the model input: 3 x ImageSize x ImageSize values.
Without a CLIP model the image side is a colour heuristic (skin-coloured and red-dominant pixel fractions) and the text side a keyword list. That fallback does not understand either modality and only flags a “safe” caption on a skin- or red-heavy image; treat it as a weak signal.
EvaluateText checks the text alone, for captions that claim to be safe while naming unsafe content.
References:
- Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP, ICML 2021)
Constructors
| Constructor | Summary |
|---|---|
TextImageAlignmentChecker(double mismatchThreshold) |
Preserves the original constructor signature using the default configuration. |
TextImageAlignmentChecker(double mismatchThreshold, IMultimodalEmbedding<T> clipModel) |
Initializes a new text-image alignment checker. |
Properties
| Property | Summary |
|---|---|
bool IsReady { get; } |
|
string ModuleName { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<SafetyFinding> Evaluate(Vector<T> content) |
|
IReadOnlyList<SafetyFinding> EvaluateAlignment(string text, Tensor<T> image) |
Evaluates alignment between text and an associated image. |
IReadOnlyList<SafetyFinding> EvaluateText(string text) |