MultimodalGuardrail<T>
Models & Types · AiDotNet.Safety.Multimodal
Unified guardrail for vision-language models (VLMs) and multimodal AI systems that validates both text and image content together.
Inheritance: ModelBase<T, Vector<T>, Vector<T>> → SafetyModuleBase<T> → MultimodalSafetyModuleBase<T> → MultimodalGuardrail<T>
For Beginners
When AI systems accept both text and images (like “describe this image”), attackers can exploit the gap between modalities. An image might be harmless by itself, and a prompt might be harmless by itself, but together they could trick the AI. This guardrail checks the combination to catch such attacks.
How It Works
Provides input/output guardrailing for multimodal systems. For text-only content, delegates to configured text safety modules. For image-only content, delegates to configured image safety modules. For combined text+image content, additionally checks for cross-modal attacks where individually safe content becomes harmful when combined.
References:
- Visual prompt injection attacks on GPT-4V (2024)
- MM-SafetyBench: Multimodal safety benchmark (2024)
- Cross-modal jailbreak attacks on multimodal LLMs (2024)
Constructors
| Constructor | Summary |
|---|---|
MultimodalGuardrail(IReadOnlyList<ITextSafetyModule<T>> textModules, IReadOnlyList<IImageSafetyModule<T>> imageModules, double crossModalThreshold) |
Initializes a new multimodal guardrail. |
Properties
| Property | Summary |
|---|---|
string ModuleName { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<SafetyFinding> EvaluateTextAudio(string text, Vector<T> audio, int sampleRate) |
|
IReadOnlyList<SafetyFinding> EvaluateTextImage(string text, Tensor<T> image) |