TextImageAlignmentChecker<T>

Models & Types · AiDotNet.Safety.Multimodal

Checks whether an image matches its caption, to catch unsafe images passed off under a safe description. With a CLIP model it compares CLIP embeddings; without one it falls back to colour heuristics.

Inheritance: ModelBase<T, Vector<T>, Vector<T>> → SafetyModuleBase<T> → TextSafetyModuleBase<T> → TextImageAlignmentChecker<T>

For Beginners

This module checks if what an image shows matches what the text says about it. For example, if someone labels an image “cute puppy” but the image shows something violent, a CLIP model sees that the picture is closer to “graphic violence” than to “cute puppy” and the module reports the mismatch.

How It Works

With a CLIP model (an IMultimodalEmbedding such as ClipNeuralNetwork, which loads the image and text encoders from ONNX files), EvaluateAlignment runs CLIP zero-shot classification (Radford et al. 2021): the image embedding is compared with the embeddings of the caption and of three unsafe-concept prompts (graphic violence, sexually explicit content, hateful symbols), the cosine similarities are scaled by the CLIP logit scale of 100 and turned into probabilities with a softmax, and a finding is reported when an unsafe concept takes at least mismatchThreshold of the probability while the caption itself does not describe unsafe content. The image must already be preprocessed to the model input: 3 x ImageSize x ImageSize values.

Without a CLIP model the image side is a colour heuristic (skin-coloured and red-dominant pixel fractions) and the text side a keyword list. That fallback does not understand either modality and only flags a “safe” caption on a skin- or red-heavy image; treat it as a weak signal.

EvaluateText checks the text alone, for captions that claim to be safe while naming unsafe content.

References:

  • Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP, ICML 2021)

Constructors

Constructor Summary
TextImageAlignmentChecker(double mismatchThreshold) Preserves the original constructor signature using the default configuration.
TextImageAlignmentChecker(double mismatchThreshold, IMultimodalEmbedding<T> clipModel) Initializes a new text-image alignment checker.

Properties

Property Summary
bool IsReady { get; }
string ModuleName { get; }

Methods

Method Summary
IReadOnlyList<SafetyFinding> Evaluate(Vector<T> content)
IReadOnlyList<SafetyFinding> EvaluateAlignment(string text, Tensor<T> image) Evaluates alignment between text and an associated image.
IReadOnlyList<SafetyFinding> EvaluateText(string text)