Skip to content

Textmate Grammar Python

Ruff Checked with ty prek CI/CD

A Python port of the vscode-textmate grammar tokenization engine. It parses TextMate grammars and tokenizes source line by line while preserving the state stack between lines.

The engine's token streams are checked against vscode-textmate 9.3.2 using a Node oracle and a differential corpus spanning MATLAB, Markdown, Python, and TypeScript.

flowchart LR
    G[TextMate grammar] --> R[Registry]
    C[source lines] --> R
    R --> T[tokens + rule stack]
    T -. differential comparison .-> O[vscode-textmate 9.3.2 oracle]
    click O "https://github.com/microsoft/vscode-textmate"

Sources