TextTensorDatasetConverter

Helpers & Utilities · AiDotNet.Agentic.SelfImproving

Tokenizes a text FineTuningDataset into tensor supervised-fine-tuning data so a tensor model (e.g. MambaLanguageModel) can be fine-tuned directly. This is the “tokenizer in front of a tensor model” step the string-only FineTuningDataConverter leaves to the model pipeline.

For Beginners

Turns “good run” text into the number grids a neural language model learns from: for every position it records “given the words so far, the next word should be this one.”

How It Works

Each example’s “prompt + completion” text is tokenized to a fixed-length window. The input is the one-hot encoding of tokens [0..L-1] and the target is the one-hot of the next tokens [1..L] — standard next-token (causal LM) supervision. Both are shape [1, L, vocab], so a language model’s per-position logits and the targets flatten to equal-length vectors for the cross-entropy loss. Each example’s reward is carried as its sample weight.

Unlike a string-typed fine-tune (where the SFT loss step cannot turn a string into a numeric vector), tensor inputs/outputs convert cleanly, so this is the path that actually trains a tensor model end-to-end.

Methods

Method Summary
FineTuningData<T, Tensor<T>, Tensor<T>> ToTensorData(FineTuningDataset dataset, IGenerationTokenizer tokenizer, int vocabSize, int sequenceLength) Converts a reward-filtered dataset to tensor next-token supervised data.