VideoCLIP<T>
Models & Types · AiDotNet.Video.Understanding
VideoCLIP model for video-text understanding and retrieval.
Inheritance: NeuralNetworkBase<T> → VideoCLIP<T>
For Beginners
VideoCLIP learns to understand both videos and text descriptions in a shared “embedding space” where similar concepts are close together.
Key capabilities:
- Video-to-Text Search: Find text descriptions that match a video
- Text-to-Video Search: Find videos that match a text query
- Zero-Shot Classification: Classify videos into categories without training
- Video Captioning: Generate descriptions for videos
- Video Question Answering: Answer questions about video content
The model creates embeddings (numerical representations) for both videos and text that can be compared using similarity measures. Videos and their corresponding descriptions will have similar embeddings.
How It Works
Technical Details:
- Contrastive learning on video-text pairs
- Temporal transformer for video understanding
- Text transformer for language understanding
- Joint embedding space with dot-product similarity
- Pre-trained on large-scale video-text datasets
Reference: Xu et al., “VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding” EMNLP 2021.
Example
// Create a VideoCLIP model for video-text understanding
var videoCLIP = new VideoCLIP<double>();
// Or configure with custom embedding dimensions
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.MultiClassClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 400);
var model = new VideoCLIP<double>(architecture, options: new VideoCLIPVideoOptions { NumFrames = 32, EmbeddingDim = 512 });
Constructors
| Constructor | Summary |
|---|---|
VideoCLIP() |
Initializes a new instance with default architecture settings (32-frame 224x224 RGB clips). |
VideoCLIP(NeuralNetworkArchitecture<T> architecture, VideoCLIPVideoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Initializes a new instance of the VideoCLIP class. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Gets whether training is supported. |
Methods
| Method | Summary |
|---|---|
List<ValueTuple<int, double>> TextToVideoRetrieval(string query, List<Tensor<T>> videoEmbeddings, int topK) |
Retrieves the most similar videos to a text query. |
List<ValueTuple<string, double>> VideoToTextRetrieval(Tensor<T> videoFrames, List<string> candidateTexts, int topK) |
Retrieves the most similar text descriptions for a video. |
List<ValueTuple<string, double>> ZeroShotClassify(Tensor<T> videoFrames, List<string> classTexts) |
Performs zero-shot video classification. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ComputeSimilarityMatrix(List<Tensor<T>> videoFramesBatch, List<Tensor<T>> textsBatch) |
Computes video-text similarity matrix for a batch. |
Tensor<T> EncodeText(Tensor<T> tokenIds) |
Encodes text into an embedding vector. |
Tensor<T> EncodeVideo(Tensor<T> videoFrames) |
Encodes a video into an embedding vector. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
double ComputeSimilarity(Tensor<T> videoEmbedding, Tensor<T> textEmbedding) |
Computes similarity between video and text embeddings. |