IVideoCLIPModel<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for VideoCLIP-style models that align video and text in a shared embedding space.
Implements: IMultimodalEmbedding<T>
Known implementations: VideoCLIPNeuralNetwork<T>
For Beginners
VideoCLIP is like CLIP but for videos!
While CLIP matches images with text, VideoCLIP matches VIDEOS with text:
- Understands actions and events that unfold over time
- Can find videos matching text descriptions
- Can generate descriptions for video clips
Key capabilities:
- Temporal understanding: “A person picks up a ball then throws it”
- Action recognition: “Playing basketball”, “Cooking”, “Dancing”
- Video retrieval: Find videos matching any text query
- Video-text alignment: Match video segments to text descriptions
Architecture differences from CLIP:
- Processes multiple frames, not just one image
- Uses temporal attention/pooling across frames
- Learns motion and action patterns
How It Works
VideoCLIP extends CLIP’s contrastive learning paradigm to the video domain, enabling text-to-video and video-to-text retrieval, action recognition, and temporal understanding.
Properties
| Property | Summary |
|---|---|
TemporalAggregationType TemporalAggregation { get; } |
Gets the temporal aggregation method used. |
double FrameRate { get; } |
Gets the frame rate (frames per second) for video sampling. |
int NumFrames { get; } |
Gets the number of frames the model processes per video clip. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> PredictNextAction(IEnumerable<Tensor<T>> frames, IEnumerable<string> possibleNextActions) |
Predicts the next action or event in a video. |
Dictionary<string, T> ZeroShotActionRecognition(IEnumerable<Tensor<T>> frames, IEnumerable<string> actionLabels) |
Performs zero-shot action classification on a video. |
IEnumerable<ValueTuple<int, T>> RetrieveTextsForVideo(IEnumerable<Tensor<T>> frames, IEnumerable<string> candidateTexts, int topK) |
Retrieves the most relevant text descriptions for a video. |
IEnumerable<ValueTuple<int, T>> RetrieveVideos(string query, IEnumerable<Vector<T>> videoEmbeddings, int topK) |
Retrieves the most relevant videos for a text query. |
IEnumerable<ValueTuple<int, int, T>> LocalizeMoments(IEnumerable<Tensor<T>> frames, string query, int windowSize) |
Localizes moments in a video that match a text description. |
IEnumerable<Vector<T>> GetVideoEmbeddings(IEnumerable<IEnumerable<Tensor<T>>> videos) |
Converts multiple videos into embedding vectors in a batch. |
T ComputeVideoTextSimilarity(string text, IEnumerable<Tensor<T>> frames) |
Computes similarity between a text description and a video. |
Tensor<T> ComputeTemporalSimilarityMatrix(IEnumerable<Tensor<T>> video1Frames, IEnumerable<Tensor<T>> video2Frames) |
Computes temporal similarity matrix between video segments. |
Tensor<T> ExtractFrameFeatures(IEnumerable<Tensor<T>> frames) |
Extracts frame-level features before temporal aggregation. |
Vector<T> GetVideoEmbedding(IEnumerable<Tensor<T>> frames) |
Converts a video (sequence of frames) into an embedding vector. |
string AnswerVideoQuestion(IEnumerable<Tensor<T>> frames, string question, int maxLength) |
Answers a question about video content. |
string GenerateVideoCaption(IEnumerable<Tensor<T>> frames, int maxLength) |
Generates a caption describing the video content. |