Tokenization Struct Reference
[Token - Tokenization module]

Tokenization. More...

Data Fields
TokenList *	tokens
const char *	process_ending

---

Detailed Description

Tokenization.

Cut-out of a reference string into a sequence of tokens

---

Field Documentation

const char* Tokenization::process_ending

Position of the last processed character. This information can be useful when a string to treat finished in the middle of a potential token (for example in the case of a streamed source).

TokenList* Tokenization::tokens

The list of tokens that forms the tokenization

---