ToonCrafter<T>
Models & Types · AiDotNet.Video.FrameInterpolation
ToonCrafter generative cartoon interpolation for large non-linear animated motion.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → ToonCrafter<T>
For Beginners
ToonCrafter is designed specifically for cartoons and animations. Unlike real-world video where objects move smoothly, cartoon characters can teleport, stretch, and move in impossible ways. This model understands these animation-specific motion patterns and generates in-between frames that look natural for animated content.
How It Works
References:
- Paper: “ToonCrafter: Generative Cartoon Interpolation” (2024)
ToonCrafter specializes in cartoon and animation frame interpolation where motions are large, non-linear, and don’t follow physical constraints. It uses a latent diffusion approach adapted for animation:
- Dual-reference conditioning: conditions the diffusion process on both start and end
cartoon frames simultaneously, using cross-attention to attend to features from both
- Sketch-guided generation: optionally uses extracted sketch/edge maps as structural
guidance, ensuring generated frames maintain the line art style and character proportions
- Toon-adapted noise schedule: a modified diffusion noise schedule that works better with
the flat colors and sharp edges typical of cartoon/animation content
- Large motion capability: handles the extreme, physically-unrealistic motions common in
animation (e.g., squash-and-stretch, sudden direction changes, exaggerated physics)
Example
// Create a ToonCrafter model for cartoon frame interpolation
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.Generative,
inputHeight: 256, inputWidth: 256, inputDepth: 3);
var options = new ToonCrafterOptions();
var toonCrafter = new ToonCrafter<double>(architecture, options);
// Or load a pre-trained ONNX model for inference
var toonCrafterOnnx = new ToonCrafter<double>(architecture, "tooncrafter_model.onnx");
Constructors
| Constructor | Summary |
|---|---|
ToonCrafter(NeuralNetworkArchitecture<T> architecture, ToonCrafterOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a ToonCrafter model for native training and inference. |
ToonCrafter(NeuralNetworkArchitecture<T> architecture, string modelPath, ToonCrafterOptions options) |
Creates a ToonCrafter model for native training and inference. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |