Lexer¶
The Lexer processes the source string by breaking it into a sequence of Tokens.
-
class
knights.lexer.TokenType¶ An Emum of token types. Valid values are:
- comment
- text
- var
- block
-
class
knights.lexer.Token¶ -
mode¶ A TokenType.
-
content¶ The raw text content of the token.
-
lineno¶ An estimate of the source line.
-
-
knights.lexer.tokenise(source)¶ A generator yielding Tokens from the source.
This uses re.finditer to break up the source string for tag, var and comments, inferring text nodes between.