GpuVectorScorer
Helpers & Utilities · AiDotNet.RetrievalAugmentedGeneration.VectorSearch
GPU-accelerated batched similarity scoring for vector search.
How It Works
This helper computes the similarity/distance between a single query vector and a large
batch of stored vectors in a single GPU matrix multiply, mirroring how FAISS-GPU /
Milvus-GPU accelerate brute-force (“flat”) search. All stored vectors are packed into an
[N, dim] row-major matrix, the query is uploaded as a [dim, 1] column, and a
single Gemm produces the [N, 1] dot-product vector. Metric-specific closed
forms then derive the final score from the dot products and precomputed L2 norms.
The GPU abstraction reused is the same one the neural-network layers use:
Current cast to
DirectGpuTensorEngine, its IsGpuAvailable probe, and the
IDirectGpuBackend returned by GetBackend() (AllocateBuffer,
Gemm, DownloadBuffer). No raw CUDA/OpenCL is written here.
Fallback contract.TryScoreBatch returns false — signalling
the caller to use its existing CPU path — whenever any of the following hold:
the AIDOTNET_DISABLE_GPU environment variable is set; no GPU engine/backend is
available; the batch is smaller than the GPU threshold (small batches are faster on the
CPU than the upload/compute/download round-trip); the metric has no GPU closed form
(only dot-product, cosine and Euclidean are accelerated); vector dimensions are
inconsistent; or any GPU error occurs. In every fallback case the caller’s CPU brute-force
path runs, so results are always correct and CI (which sets AIDOTNET_DISABLE_GPU)
exercises the CPU path deterministically.
The types used here (DirectGpuTensorEngine, IDirectGpuBackend)
ship in the AiDotNet.Tensors package for every target framework (net10.0, net8.0, net471),
so this file compiles on all TFMs; the GPU path simply stays dormant at runtime when no
device is present.
Properties
| Property | Summary |
|---|---|
bool IsGpuDisabledByEnvironment { get; } |
Returns true when the AIDOTNET_DISABLE_GPU environment variable is set to a non-empty, non-“0”/“false” value. |
Methods
| Method | Summary |
|---|---|
bool CanAccelerate(ISimilarityMetric<T> metric, int count, int gpuThreshold) |
Returns true when a batch of the given size with the given metric would be routed to the GPU (metric supported, batch at/above threshold, and a GPU device available). |
bool IsGpuAvailable() |
Returns true when a GPU tensor engine is present and enabled (and not disabled via the environment variable). |
bool TryScoreBatch(ISimilarityMetric<T> metric, Vector<T> query, IReadOnlyList<Vector<T>> docs, int gpuThreshold, T[] scores) |
Scores query against every vector in docs on the GPU in a single batched matrix multiply, producing one score per document in the same order as docs and using the same semantics as the CPU metric. |
Fields
| Field | Summary |
|---|---|
int DefaultGpuThreshold |
Default minimum number of stored vectors before the GPU path is attempted. |