GpuVectorScorer

Helpers & Utilities · AiDotNet.RetrievalAugmentedGeneration.VectorSearch

GPU-accelerated batched similarity scoring for vector search.

How It Works

This helper computes the similarity/distance between a single query vector and a large batch of stored vectors in a single GPU matrix multiply, mirroring how FAISS-GPU / Milvus-GPU accelerate brute-force (“flat”) search. All stored vectors are packed into an [N, dim] row-major matrix, the query is uploaded as a [dim, 1] column, and a single Gemm produces the [N, 1] dot-product vector. Metric-specific closed forms then derive the final score from the dot products and precomputed L2 norms.

The GPU abstraction reused is the same one the neural-network layers use: Current cast to DirectGpuTensorEngine, its IsGpuAvailable probe, and the IDirectGpuBackend returned by GetBackend() (AllocateBuffer, Gemm, DownloadBuffer). No raw CUDA/OpenCL is written here.

Fallback contract.TryScoreBatch returns false — signalling the caller to use its existing CPU path — whenever any of the following hold: the AIDOTNET_DISABLE_GPU environment variable is set; no GPU engine/backend is available; the batch is smaller than the GPU threshold (small batches are faster on the CPU than the upload/compute/download round-trip); the metric has no GPU closed form (only dot-product, cosine and Euclidean are accelerated); vector dimensions are inconsistent; or any GPU error occurs. In every fallback case the caller’s CPU brute-force path runs, so results are always correct and CI (which sets AIDOTNET_DISABLE_GPU) exercises the CPU path deterministically.

The types used here (DirectGpuTensorEngine, IDirectGpuBackend) ship in the AiDotNet.Tensors package for every target framework (net10.0, net8.0, net471), so this file compiles on all TFMs; the GPU path simply stays dormant at runtime when no device is present.

Properties

Property Summary
bool IsGpuDisabledByEnvironment { get; } Returns true when the AIDOTNET_DISABLE_GPU environment variable is set to a non-empty, non-“0”/“false” value.

Methods

Method Summary
bool CanAccelerate(ISimilarityMetric<T> metric, int count, int gpuThreshold) Returns true when a batch of the given size with the given metric would be routed to the GPU (metric supported, batch at/above threshold, and a GPU device available).
bool IsGpuAvailable() Returns true when a GPU tensor engine is present and enabled (and not disabled via the environment variable).
bool TryScoreBatch(ISimilarityMetric<T> metric, Vector<T> query, IReadOnlyList<Vector<T>> docs, int gpuThreshold, T[] scores) Scores query against every vector in docs on the GPU in a single batched matrix multiply, producing one score per document in the same order as docs and using the same semantics as the CPU metric.

Fields

Field Summary
int DefaultGpuThreshold Default minimum number of stored vectors before the GPU path is attempted.