NerTextEncoder
Models & Types · AiDotNet.NER.Preprocessing
Converts raw text into the packed integer indices consumed by
WordCharEmbeddingLayer /
WordCharBiLSTMCRF, and builds the word and
character vocabularies. This is the data-layer counterpart to the model, mirroring the standard
NER pipeline (a separate vocabulary/tokenizer feeding a model that owns its embeddings).
For Beginners
models work on numbers, not text. This helper assigns every word and every letter a fixed number, splits punctuation correctly, and packs each sentence into a grid of numbers the model can read. “Unknown” words it never saw still get a consistent number, so the model behaves the same every run.
How It Works
Responsibilities:
- Tokenization: a regex word tokenizer that splits punctuation off words, so “Paris,”
becomes [“Paris”, “,”] instead of an out-of-vocabulary blob — fixing the recall loss a naive whitespace split causes.
- Vocabulary construction: a case-folded word vocabulary (words are lowercased for the
word-embedding lookup, matching GloVe) and a case-preserving character vocabulary (so capitalization is captured by the character encoder, not discarded).
- Encoding: a sentence becomes a
[sequenceLength, 1 + maxWordLength]integer
matrix: column 0 is the word id, the rest are the word’s character ids (zero-padded).
Index 0 is the padding id and index 1 is the unknown id in both vocabularies, so unseen words/chars map deterministically to a stable “[UNK]” embedding rather than hash-derived noise.
Properties
| Property | Summary |
|---|---|
Vocabulary CharVocabulary { get; } |
Gets the character vocabulary (case-preserving). |
Vocabulary WordVocabulary { get; } |
Gets the word vocabulary (word ids are looked up case-folded). |
int CharVocabSize { get; } |
Gets the character vocabulary size (including [PAD] and [UNK]). |
int MaxWordLength { get; } |
Gets the maximum number of characters encoded per word; longer words are truncated. |
int WordVocabSize { get; } |
Gets the word vocabulary size (including [PAD] and [UNK]). |
Methods
| Method | Summary |
|---|---|
NerTextEncoder Build(IEnumerable<string[]> tokenizedSentences, int maxWordLength) |
Builds a NerTextEncoder from already-tokenized training sentences, populating the word and character vocabularies. |
double[] EncodePacked(string[] tokens, int sequenceLength) |
Encodes a tokenized sentence into a packed [sequenceLength, 1 + MaxWordLength] integer matrix as Double values: column 0 is the (case-folded) word id, columns 1.. |
string[] Tokenize(string sentence) |
Splits a sentence into word/punctuation tokens, separating trailing punctuation from words. |
Fields
| Field | Summary |
|---|---|
string PadToken |
The padding token, reserved at index 0 in both vocabularies. |
string UnkToken |
The unknown token, reserved at index 1 in both vocabularies. |