DevicePagedKVCache
Models & Types · AiDotNet.Tensors.Engines.DirectGpu
No summary documentation available yet.
Implements: IDisposable
Constructors
| Constructor | Summary |
|---|---|
DevicePagedKVCache(IDirectGpuBackend backend, int maxBlocks, int blockSize, int heads, int headDim) |
Properties
| Property | Summary |
|---|---|
IGpuBuffer KeyBlocks { get; } |
|
IGpuBuffer ValueBlocks { get; } |
|
int AllocatedBlocks { get; } |
|
int BlockSize { get; } |
|
int HeadDim { get; } |
|
int Heads { get; } |
|
int MaxBlocks { get; } |
Methods
| Method | Summary |
|---|---|
IGpuBuffer GetBlockTableBuffer(int seqId) |
|
IReadOnlyList<int> GetBlockTable(int seqId) |
|
int GetLength(int seqId) |
|
void Append(int seqId, IGpuBuffer newKeys, IGpuBuffer newValues, int newLen) |
|
void Append(int seqId, float[] newKeys, float[] newValues) |
|
void Dispose() |
|
void Free(int seqId) |
|
void ShareBlocks(int sourceSeqId, int targetSeqId, int prefixLen) |