IPAdapterModel<T>
Models & Types · AiDotNet.Diffusion.Control
IP-Adapter model for image-based prompt conditioning in diffusion models.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → IPAdapterModel<T>
For Beginners
IP-Adapter lets you use pictures as instructions for the AI instead of just text.
Think of it like:
- Showing someone a photo and saying “make something like this”
- The AI extracts the style, composition, and content from your image
- It then applies those elements to create new images
Use cases:
- Style transfer: “Generate in the style of this artwork”
- Face preservation: Keep a person’s likeness in different scenes
- Object consistency: Maintain the same object across images
- Scene composition: Use reference for layout/arrangement
Key advantage: Combines with text prompts for precise control
How It Works
IP-Adapter (Image Prompt Adapter) enables using reference images as prompts to guide image generation. It decouples cross-attention for text and image features, allowing fine-grained control over image style, composition, and content transfer.
Technical details:
- Uses a pretrained image encoder (like CLIP ViT)
- Projects image features to text embedding space
- Injects via decoupled cross-attention mechanism
- Supports multiple reference images (multi-IP)
- Adjustable image prompt weight (0-1)
Reference: Ye et al., “IP-Adapter: Text Compatible Image Prompt Adapter”, 2023
Example
// Create an IP-Adapter model
var ipAdapter = new IPAdapterModel<float>();
// Generate with image reference
var referenceImage = ImageHelper<float>.LoadImage("style_reference.png");
var image = ipAdapter.GenerateWithImagePrompt(
textPrompt: "A beautiful landscape",
imagePrompt: referenceImage,
imagePromptWeight: 0.7);
// Multi-image reference
var faceImage = ImageHelper<float>.LoadImage("face.png");
var styleImage = ImageHelper<float>.LoadImage("art_style.png");
var composed = ipAdapter.GenerateWithMultiImagePrompt(
textPrompt: "Portrait painting",
imagePrompts: new[] { faceImage, styleImage },
imageWeights: new[] { 0.8, 0.5 });
Constructors
| Constructor | Summary |
|---|---|
IPAdapterModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> baseUNet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int embedDim, int? seed) |
Initializes a new instance of IPAdapterModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
double ImagePromptWeight { get; set; } |
Gets or sets the default image prompt weight (0-1). |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GenerateWithImagePrompt(string textPrompt, Tensor<T> imagePrompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, double? imagePromptWeight, int? seed) |
Generates an image with image prompt conditioning. |
Tensor<T> GenerateWithMultiImagePrompt(string textPrompt, Tensor<T>[] imagePrompts, double[] imageWeights, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) |
Generates an image with multiple image prompts. |