Pix2Pix<T>

Models & Types · AiDotNet.NeuralNetworks

Represents a Pix2Pix GAN for paired image-to-image translation tasks.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → ImageTranslationModelLayoutBase<T> → Pix2Pix<T>

For Beginners

Pix2Pix transforms one type of image to another.

Key features:

  • Learns from paired examples (input A becomes output B)
  • Generator: U-Net architecture preserves spatial information
  • Discriminator: PatchGAN focuses on local image patches
  • Loss: Both “looks real” and “matches input”

Example use cases:

  • Convert sketches to realistic photos
  • Colorize black-and-white images
  • Transform day scenes to night
  • Semantic labels to photorealistic images
  • Map to satellite image

Reference: Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks” (2017)

How It Works

Pix2Pix is a conditional GAN for paired image-to-image translation:

  • Uses a U-Net generator with skip connections
  • Uses a PatchGAN discriminator that classifies image patches
  • Combines adversarial loss with L1 reconstruction loss
  • Requires paired training data (input-output pairs)
  • Works for various tasks: edges to photo, day to night, sketch to image, etc.

Example

var options = new Pix2PixOptions { };
var input = Tensor<float>.CreateRandom(new[] { 1, 3, 256, 256 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
    var generatorArchitecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 8);
    var discriminatorArchitecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 1);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new Pix2Pix<float>(
        generatorArchitecture, discriminatorArchitecture,
        inputType: InputType.ThreeDimensional))
    .Build(trainX, trainY);
var translated = result.Predict(input);

Constructors

Constructor Summary
Pix2Pix(NeuralNetworkArchitecture<T> generatorArchitecture, NeuralNetworkArchitecture<T> discriminatorArchitecture, InputType inputType, Pix2PixOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> generatorOptimizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> discriminatorOptimizer, ILossFunction<T> lossFunction) Initializes a new instance of the Pix2Pix class.

Properties

Property Summary
NeuralNetworkBase<T> Discriminator { get; set; } Gets the PatchGAN discriminator network.
NeuralNetworkBase<T> Generator { get; set; } Gets the U-Net generator network.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
ModelMetadata<T> GetModelMetadata()
Tensor<T> Translate(Tensor<T> inputImages) Translates input images to output images.
ValueTuple<T, T, T> TrainStep(Tensor<T> inputImages, Tensor<T> targetImages) Performs one training step for Pix2Pix.
void ResetOptimizerState() Resets both optimizer states for a fresh training run.