IVideoCLIPModel<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for VideoCLIP-style models that align video and text in a shared embedding space.

Implements: IMultimodalEmbedding<T>

Known implementations: VideoCLIPNeuralNetwork<T>

For Beginners

VideoCLIP is like CLIP but for videos!

While CLIP matches images with text, VideoCLIP matches VIDEOS with text:

  • Understands actions and events that unfold over time
  • Can find videos matching text descriptions
  • Can generate descriptions for video clips

Key capabilities:

  • Temporal understanding: “A person picks up a ball then throws it”
  • Action recognition: “Playing basketball”, “Cooking”, “Dancing”
  • Video retrieval: Find videos matching any text query
  • Video-text alignment: Match video segments to text descriptions

Architecture differences from CLIP:

  • Processes multiple frames, not just one image
  • Uses temporal attention/pooling across frames
  • Learns motion and action patterns

How It Works

VideoCLIP extends CLIP’s contrastive learning paradigm to the video domain, enabling text-to-video and video-to-text retrieval, action recognition, and temporal understanding.

Properties

Property Summary
TemporalAggregationType TemporalAggregation { get; } Gets the temporal aggregation method used.
double FrameRate { get; } Gets the frame rate (frames per second) for video sampling.
int NumFrames { get; } Gets the number of frames the model processes per video clip.

Methods

Method Summary
Dictionary<string, T> PredictNextAction(IEnumerable<Tensor<T>> frames, IEnumerable<string> possibleNextActions) Predicts the next action or event in a video.
Dictionary<string, T> ZeroShotActionRecognition(IEnumerable<Tensor<T>> frames, IEnumerable<string> actionLabels) Performs zero-shot action classification on a video.
IEnumerable<ValueTuple<int, T>> RetrieveTextsForVideo(IEnumerable<Tensor<T>> frames, IEnumerable<string> candidateTexts, int topK) Retrieves the most relevant text descriptions for a video.
IEnumerable<ValueTuple<int, T>> RetrieveVideos(string query, IEnumerable<Vector<T>> videoEmbeddings, int topK) Retrieves the most relevant videos for a text query.
IEnumerable<ValueTuple<int, int, T>> LocalizeMoments(IEnumerable<Tensor<T>> frames, string query, int windowSize) Localizes moments in a video that match a text description.
IEnumerable<Vector<T>> GetVideoEmbeddings(IEnumerable<IEnumerable<Tensor<T>>> videos) Converts multiple videos into embedding vectors in a batch.
T ComputeVideoTextSimilarity(string text, IEnumerable<Tensor<T>> frames) Computes similarity between a text description and a video.
Tensor<T> ComputeTemporalSimilarityMatrix(IEnumerable<Tensor<T>> video1Frames, IEnumerable<Tensor<T>> video2Frames) Computes temporal similarity matrix between video segments.
Tensor<T> ExtractFrameFeatures(IEnumerable<Tensor<T>> frames) Extracts frame-level features before temporal aggregation.
Vector<T> GetVideoEmbedding(IEnumerable<Tensor<T>> frames) Converts a video (sequence of frames) into an embedding vector.
string AnswerVideoQuestion(IEnumerable<Tensor<T>> frames, string question, int maxLength) Answers a question about video content.
string GenerateVideoCaption(IEnumerable<Tensor<T>> frames, int maxLength) Generates a caption describing the video content.