DevicePagedKVCache

Models & Types · AiDotNet.Tensors.Engines.DirectGpu

No summary documentation available yet.

Implements: IDisposable

Constructors

Constructor Summary
DevicePagedKVCache(IDirectGpuBackend backend, int maxBlocks, int blockSize, int heads, int headDim)

Properties

Property Summary
IGpuBuffer KeyBlocks { get; }
IGpuBuffer ValueBlocks { get; }
int AllocatedBlocks { get; }
int BlockSize { get; }
int HeadDim { get; }
int Heads { get; }
int MaxBlocks { get; }

Methods

Method Summary
IGpuBuffer GetBlockTableBuffer(int seqId)
IReadOnlyList<int> GetBlockTable(int seqId)
int GetLength(int seqId)
void Append(int seqId, IGpuBuffer newKeys, IGpuBuffer newValues, int newLen)
void Append(int seqId, float[] newKeys, float[] newValues)
void Dispose()
void Free(int seqId)
void ShareBlocks(int sourceSeqId, int targetSeqId, int prefixLen)