VideoCLIPNeuralNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

VideoCLIP neural network for video-text alignment and temporal understanding.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → VideoCLIPNeuralNetwork<T>

Implements: IVideoCLIPModel<T>

For Beginners

VideoCLIP is like CLIP but for videos!

Architecture overview:

  1. Vision Encoder: Extracts features from each frame (shared CLIP ViT)
  2. Temporal Encoder: Aggregates frame features over time
  3. Text Encoder: Processes text descriptions
  4. Contrastive Learning: Aligns video and text in shared embedding space

Key capabilities:

  • Video retrieval: Find videos matching text descriptions
  • Action recognition: Classify actions without training
  • Moment localization: Find specific moments in videos
  • Video QA: Answer questions about video content

How It Works

VideoCLIP extends CLIP’s contrastive learning paradigm to the video domain, enabling text-to-video and video-to-text retrieval, action recognition, and temporal understanding.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var video = Tensor<float>.CreateRandom(new[] { 1, 3, 8, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new VideoCLIPNeuralNetwork<float>(architecture))
    .Build(trainX, trainY);
var embedding = result.Predict(video);

Constructors

Constructor Summary
VideoCLIPNeuralNetwork(NeuralNetworkArchitecture<T> architecture, VideoCLIPOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a VideoCLIP network using native library layers.
VideoCLIPNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string videoEncoderPath, string textEncoderPath, ITokenizer tokenizer, VideoCLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a VideoCLIP network using pretrained ONNX models.

Properties

Property Summary
TemporalAggregationType TemporalAggregation { get; }
double FrameRate { get; }
int EmbeddingDimension { get; }
int ImageSize { get; }
int MaxSequenceLength { get; }
int NumFrames { get; }

Methods

Method Summary
Dictionary<string, T> PredictNextAction(IEnumerable<Tensor<T>> frames, IEnumerable<string> possibleNextActions)
Dictionary<string, T> ZeroShotActionRecognition(IEnumerable<Tensor<T>> frames, IEnumerable<string> actionLabels)
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels)
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels)
IEnumerable<ValueTuple<int, T>> RetrieveTextsForVideo(IEnumerable<Tensor<T>> frames, IEnumerable<string> candidateTexts, int topK)
IEnumerable<ValueTuple<int, T>> RetrieveVideos(string query, IEnumerable<Vector<T>> videoEmbeddings, int topK)
IEnumerable<ValueTuple<int, int, T>> LocalizeMoments(IEnumerable<Tensor<T>> frames, string query, int windowSize)
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images)
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts)
IEnumerable<Vector<T>> GetVideoEmbeddings(IEnumerable<IEnumerable<Tensor<T>>> videos)
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch)
Matrix<T> EncodeTextBatch(IEnumerable<string> texts)
ModelMetadata<T> GetModelMetadata()
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding)
T ComputeVideoTextSimilarity(string text, IEnumerable<Tensor<T>> frames)
Tensor<T> ComputeTemporalSimilarityMatrix(IEnumerable<Tensor<T>> video1Frames, IEnumerable<Tensor<T>> video2Frames)
Tensor<T> ExtractFrameFeatures(IEnumerable<Tensor<T>> frames)
Tensor<T> ForwardForTraining(Tensor<T> input)
Vector<T> EncodeImage(double[] imageData)
Vector<T> EncodeText(string text)
Vector<T> GetImageEmbedding(Tensor<T> image)
Vector<T> GetTextEmbedding(string text)
Vector<T> GetVideoEmbedding(IEnumerable<Tensor<T>> frames)
string AnswerVideoQuestion(IEnumerable<Tensor<T>> frames, string question, int maxLength)
string GenerateVideoCaption(IEnumerable<Tensor<T>> frames, int maxLength)