VideoCLIPNeuralNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
VideoCLIP neural network for video-text alignment and temporal understanding.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → VideoCLIPNeuralNetwork<T>
Implements: IVideoCLIPModel<T>
For Beginners
VideoCLIP is like CLIP but for videos!
Architecture overview:
- Vision Encoder: Extracts features from each frame (shared CLIP ViT)
- Temporal Encoder: Aggregates frame features over time
- Text Encoder: Processes text descriptions
- Contrastive Learning: Aligns video and text in shared embedding space
Key capabilities:
- Video retrieval: Find videos matching text descriptions
- Action recognition: Classify actions without training
- Moment localization: Find specific moments in videos
- Video QA: Answer questions about video content
How It Works
VideoCLIP extends CLIP’s contrastive learning paradigm to the video domain, enabling text-to-video and video-to-text retrieval, action recognition, and temporal understanding.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var video = Tensor<float>.CreateRandom(new[] { 1, 3, 8, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new VideoCLIPNeuralNetwork<float>(architecture))
.Build(trainX, trainY);
var embedding = result.Predict(video);
Constructors
| Constructor | Summary |
|---|---|
VideoCLIPNeuralNetwork(NeuralNetworkArchitecture<T> architecture, VideoCLIPOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a VideoCLIP network using native library layers. |
VideoCLIPNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string videoEncoderPath, string textEncoderPath, ITokenizer tokenizer, VideoCLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a VideoCLIP network using pretrained ONNX models. |
Properties
| Property | Summary |
|---|---|
TemporalAggregationType TemporalAggregation { get; } |
|
double FrameRate { get; } |
|
int EmbeddingDimension { get; } |
|
int ImageSize { get; } |
|
int MaxSequenceLength { get; } |
|
int NumFrames { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> PredictNextAction(IEnumerable<Tensor<T>> frames, IEnumerable<string> possibleNextActions) |
|
Dictionary<string, T> ZeroShotActionRecognition(IEnumerable<Tensor<T>> frames, IEnumerable<string> actionLabels) |
|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels) |
|
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels) |
|
IEnumerable<ValueTuple<int, T>> RetrieveTextsForVideo(IEnumerable<Tensor<T>> frames, IEnumerable<string> candidateTexts, int topK) |
|
IEnumerable<ValueTuple<int, T>> RetrieveVideos(string query, IEnumerable<Vector<T>> videoEmbeddings, int topK) |
|
IEnumerable<ValueTuple<int, int, T>> LocalizeMoments(IEnumerable<Tensor<T>> frames, string query, int windowSize) |
|
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images) |
|
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts) |
|
IEnumerable<Vector<T>> GetVideoEmbeddings(IEnumerable<IEnumerable<Tensor<T>>> videos) |
|
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch) |
|
Matrix<T> EncodeTextBatch(IEnumerable<string> texts) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding) |
|
T ComputeVideoTextSimilarity(string text, IEnumerable<Tensor<T>> frames) |
|
Tensor<T> ComputeTemporalSimilarityMatrix(IEnumerable<Tensor<T>> video1Frames, IEnumerable<Tensor<T>> video2Frames) |
|
Tensor<T> ExtractFrameFeatures(IEnumerable<Tensor<T>> frames) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Vector<T> EncodeImage(double[] imageData) |
|
Vector<T> EncodeText(string text) |
|
Vector<T> GetImageEmbedding(Tensor<T> image) |
|
Vector<T> GetTextEmbedding(string text) |
|
Vector<T> GetVideoEmbedding(IEnumerable<Tensor<T>> frames) |
|
string AnswerVideoQuestion(IEnumerable<Tensor<T>> frames, string question, int maxLength) |
|
string GenerateVideoCaption(IEnumerable<Tensor<T>> frames, int maxLength) |