Textmate Grammar Python¶
A Python port of the vscode-textmate grammar tokenization engine. It parses TextMate grammars and tokenizes source line by line while preserving the state stack between lines.
The engine's token streams are checked against vscode-textmate 9.3.2 using a Node oracle and a differential corpus spanning MATLAB, Markdown, Python, and TypeScript.
flowchart LR
G[TextMate grammar] --> R[Registry]
C[source lines] --> R
R --> T[tokens + rule stack]
T -. differential comparison .-> O[vscode-textmate 9.3.2 oracle]
click O "https://github.com/microsoft/vscode-textmate"