ModelScopeT2VModel<T>
Models & Types · AiDotNet.Diffusion.Video
ModelScope Text-to-Video model with temporal U-Net for short video clip generation.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → ModelScopeT2VModel<T>
For Beginners
ModelScope T2V generates short video clips from text prompts.
How ModelScope T2V works:
- Text prompt is encoded by CLIP into 1024-dim embeddings
- Each video frame is encoded by the SD VAE into 4 latent channels
- The temporal U-Net processes latent frames with temporal attention and convolution
- Temporal attention ensures frames are consistent with each other
- The VAE decodes each latent frame back to pixel space
Key characteristics:
- Based on SD 1.5 architecture with temporal blocks added
- Trained on WebVid-10M dataset (10M video-text pairs)
- 256x256 base resolution with cascaded upscaling to 512x512
- 16 frames at 8 FPS by default (~2 seconds)
- One of the first open-source text-to-video models
When to use ModelScope T2V:
- Simple text-to-video generation
- Research on temporal attention mechanisms
- Lightweight video generation on modest hardware
- Building on established SD 1.5 ecosystem
Limitations:
- Low resolution (256-512px)
- Short duration (16 frames)
- Quality below modern video models
- Single temporal attention layer limits motion coherence
How It Works
ModelScope T2V extends the Stable Diffusion U-Net architecture with temporal convolution and temporal attention modules, enabling text-to-video generation. It was one of the first open-source text-to-video models trained on the WebVid-10M dataset.
Architecture components:
- Video U-Net based on SD 1.5 with temporal extension blocks
- 320 base channels with [1, 2, 4, 4] channel multipliers
- 1 temporal attention layer per block for inter-frame consistency
- CLIP text encoder for 1024-dim cross-attention conditioning
- Standard SD VAE for per-frame spatial compression (4 latent channels)
- DDPM noise schedule with scaled linear beta
Technical specifications:
- Architecture: Video U-Net (SD 1.5 + temporal blocks)
- Base channels: 320, multipliers [1, 2, 4, 4]
- Attention heads: 8
- ResNet blocks per level: 2
- Temporal attention layers: 1 per block
- Latent channels: 4 (standard SD VAE)
- Cross-attention dimension: 1024 (CLIP)
- Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
- Default: 16 frames at 8 FPS (~2 seconds)
- Training dataset: WebVid-10M
Reference: Wang et al., “ModelScope Text-to-Video Technical Report”, 2023
Example
// Create with industry-standard defaults
var modelScope = new ModelScopeT2VModel<float>();
// Generate video from text
var video = modelScope.GenerateFromText(
prompt: "A teddy bear painting a portrait",
width: 256,
height: 256,
numFrames: 16,
fps: 8,
numInferenceSteps: 50,
guidanceScale: 9.0);
Constructors
| Constructor | Summary |
|---|---|
ModelScopeT2VModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUNet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) |
Initializes a new instance of ModelScopeT2VModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsImageToVideo { get; } |
|
bool SupportsTextToVideo { get; } |
|
bool SupportsVideoToVideo { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |