IPAdapterModel<T>

Models & Types · AiDotNet.Diffusion.Control

IP-Adapter model for image-based prompt conditioning in diffusion models.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → IPAdapterModel<T>

For Beginners

IP-Adapter lets you use pictures as instructions for the AI instead of just text.

Think of it like:

  • Showing someone a photo and saying “make something like this”
  • The AI extracts the style, composition, and content from your image
  • It then applies those elements to create new images

Use cases:

  • Style transfer: “Generate in the style of this artwork”
  • Face preservation: Keep a person’s likeness in different scenes
  • Object consistency: Maintain the same object across images
  • Scene composition: Use reference for layout/arrangement

Key advantage: Combines with text prompts for precise control

How It Works

IP-Adapter (Image Prompt Adapter) enables using reference images as prompts to guide image generation. It decouples cross-attention for text and image features, allowing fine-grained control over image style, composition, and content transfer.

Technical details:

  • Uses a pretrained image encoder (like CLIP ViT)
  • Projects image features to text embedding space
  • Injects via decoupled cross-attention mechanism
  • Supports multiple reference images (multi-IP)
  • Adjustable image prompt weight (0-1)

Reference: Ye et al., “IP-Adapter: Text Compatible Image Prompt Adapter”, 2023

Example

// Create an IP-Adapter model
var ipAdapter = new IPAdapterModel<float>();
            
// Generate with image reference
var referenceImage = ImageHelper<float>.LoadImage("style_reference.png");
var image = ipAdapter.GenerateWithImagePrompt(
    textPrompt: "A beautiful landscape",
    imagePrompt: referenceImage,
    imagePromptWeight: 0.7);
            
// Multi-image reference
var faceImage = ImageHelper<float>.LoadImage("face.png");
var styleImage = ImageHelper<float>.LoadImage("art_style.png");
var composed = ipAdapter.GenerateWithMultiImagePrompt(
    textPrompt: "Portrait painting",
    imagePrompts: new[] { faceImage, styleImage },
    imageWeights: new[] { 0.8, 0.5 });

Constructors

Constructor Summary
IPAdapterModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> baseUNet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int embedDim, int? seed) Initializes a new instance of IPAdapterModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
double ImagePromptWeight { get; set; } Gets or sets the default image prompt weight (0-1).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateWithImagePrompt(string textPrompt, Tensor<T> imagePrompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, double? imagePromptWeight, int? seed) Generates an image with image prompt conditioning.
Tensor<T> GenerateWithMultiImagePrompt(string textPrompt, Tensor<T>[] imagePrompts, double[] imageWeights, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) Generates an image with multiple image prompts.