GgufModelSource

Models & Types · AiDotNet.ModelLoading.Pretrained

Opens a GGUF (llama.cpp) checkpoint as a decoder-ready source: it reads the architecture from the GGUF metadata into a HuggingFaceConfig and exposes the weights under the Hugging Face parameter names LlamaModelBuilder expects, translating each read to the GGUF tensor name (token_embd.weight, blk.N.attn_q.weight, …).

Implements: IDisposable, INamedTensorSource

For Beginners

GGUF is the format llama.cpp uses. This class understands a GGUF file well enough to rebuild the same model inside AiDotNet and serve it.

How It Works

GGUF stores linear weights [out, in] row-major — identical to Hugging Face safetensors — so no re-layout is needed; the builder’s usual [out,in]→[in,out] transpose applies unchanged. Quantized tensors (Q4_0/Q4_1/Q8_0/Q4_K/Q6_K) and F16/F32 are dequantized on read by GgufFile. Dispose the source to close the file handle.

Properties

Property Summary
HuggingFaceConfig Config { get; } The architecture read from the GGUF metadata.
IReadOnlyCollection<string> TensorNames { get; }

Methods

Method Summary
GgufModelSource Open(string path) Opens a GGUF file and prepares its config + Hugging Face name mapping.
ITokenizer BuildTokenizer() Builds the model’s tokenizer from the GGUF tokenizer.ggml.* metadata: the vocabulary (tokens), BPE merge ranks (merges), added/control tokens (token_type), and the BOS/EOS/UNK/PAD ids.
bool TryReadQ8_0(string hfName, sbyte[] qs, float[] scales) Reads a directly-mapped tensor’s native Q8_0 blocks (int8 payload + one fp32 scale per 32-value block) WITHOUT dequantizing, resolving the Hugging-Face hfName to its GGUF name.
double[] ReadAsDouble(string name)
void Dispose()