ProgramTokenComplexityDescriptor

Models & Types · AiDotNet.Evolution.Programs

Measures a candidate program by the number of lexical tokens in its normalized source.

Implements: IProgramDescriptor

For Beginners

This descriptor asks “how many pieces is this program made of?” — where a piece is a name, a number, a piece of text in quotes, or a symbol such as a bracket or plus sign. It is a better measure of how big a program really is than counting characters, because renaming a variable to something longer does not change it. Use it as one axis of a quality-diversity grid to keep both compact and elaborate solutions alive in the archive.

How It Works

Counting tokens rather than characters gives an archive axis that tracks structure instead of typography. A model that renames i to rowIndex, wraps a long line, or adds indentation changes the character count substantially and the token count not at all, so candidates stay in the cell their structure earns. The tokenizer is language neutral and deterministic: it groups runs of letters, digits, underscores, and dots into one identifier or number token, so self.value and 3.14 each count once; it treats a quoted run as a single string token; and it counts every other non-white-space character as its own punctuation token.

Comments are counted like any other text because stripping them correctly would require a per-language parser, and a silent, partially correct comment stripper would make the axis depend on which language guess was made. The count is therefore an honest proxy for source size rather than an estimate of semantic complexity.

Constructors

Constructor Summary
ProgramTokenComplexityDescriptor(string name) Initializes a token-count descriptor.

Properties

Property Summary
string Name { get; }

Methods

Method Summary
double Compute(ProgramGenome genome)
int CountTokens(string source) Counts the lexical tokens in a source snippet using this descriptor’s tokenizer.

Fields

Field Summary
string DefaultName The descriptor name used when none is supplied.