Skip to content

API reference

The public API mirrors vscode-textmate's registry and stateful tokenization model.

textmate_grammar

Python port of vscode-textmate's grammar tokenizer.

Modules:

  • engine –

    Public vscode-textmate-shaped tokenization API.

  • parsers –

Classes:

  • Grammar –

    A loaded grammar and its vscode-textmate line tokenizer.

  • Registry –

    Registry options, dependency resolution, and lazy grammar construction.

  • RegistryOptions –

    Synchronous counterpart of upstream RegistryOptions.

  • SyncRegistry –

    Synchronous equivalent of upstream SyncRegistry.

Functions:

  • create_grammar –

    Construct a grammar; external_grammar is the registry integration seam.

  • parse_raw_grammar –

    Parse a grammar from content (parseRawGrammar.ts upstream).

Grammar

Grammar(
    scope_name: str,
    grammar: Mapping[str, Any],
    initial_language: int = 1,
    embedded_languages: Mapping[str, int] | None = None,
    token_types: Mapping[str, int] | None = None,
    balanced_bracket_selectors: BalancedBracketSelectors
    | None = None,
    grammar_repository: Any = None,
    onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
    *,
    external_grammar: Callable[
        [str], Mapping[str, Any] | None
    ]
    | None = None,
)

              flowchart TD
              textmate_grammar.Grammar[Grammar]
              textmate_grammar.engine.rules.RuleRegistry[RuleRegistry]
              textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
              textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
              textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]

                              textmate_grammar.engine.rules.RuleRegistry --> textmate_grammar.Grammar
                                textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.rules.RuleRegistry
                                textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                
                textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                


                textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.Grammar
                                textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                
                textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                



              click textmate_grammar.Grammar href "" "textmate_grammar.Grammar"
              click textmate_grammar.engine.rules.RuleRegistry href "" "textmate_grammar.engine.rules.RuleRegistry"
              click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
              click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
              click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
            

A loaded grammar and its vscode-textmate line tokenizer.

Methods:

  • create_onig_scanner –

    Port of Grammar.createOnigScanner (grammar.ts:177-179).

  • create_onig_string –

    Port of Grammar.createOnigString (grammar.ts:180-181).

  • get_rule –

    Port of Grammar.getRule (grammar.ts:252-254). May return the

  • register_rule –

    Port of Grammar.registerRule (grammar.ts:245-250).

create_onig_scanner

create_onig_scanner(sources: Sequence[str]) -> Any

Port of Grammar.createOnigScanner (grammar.ts:177-179).

create_onig_string

create_onig_string(string: str) -> Any

Port of Grammar.createOnigString (grammar.ts:180-181).

get_rule

get_rule(rule_id: int) -> Rule

Port of Grammar.getRule (grammar.ts:252-254). May return the (python) None of a not-yet-registered rule while its factory is still running — upstream yields undefined there, and the only reader that can act on it treats non-None like undefined (see RuleFactory.compile_patterns).

register_rule

register_rule(factory: Callable[[int], T]) -> T

Port of Grammar.registerRule (grammar.ts:245-250).

Registry

Registry(
    options: RegistryOptions,
    *,
    grammar_factory: GrammarFactory | None = None,
)

Registry options, dependency resolution, and lazy grammar construction.

Methods:

  • addGrammar –

    Register and construct a grammar, matching upstream's convenience method.

  • loadGrammar –

    vscode-textmate spelling of :meth:load_grammar.

  • load_grammar –

    Load a grammar and every referenced grammar before constructing it.

  • register_grammar –

    Register an already parsed raw grammar and optional injection scopes.

addGrammar

addGrammar(
    raw_grammar: IRawGrammar,
    injections: Sequence[str] = (),
    initial_language: int = 0,
    embedded_languages: Mapping[str, int] | None = None,
) -> TokenizerLike

Register and construct a grammar, matching upstream's convenience method.

loadGrammar

loadGrammar(
    initial_scope_name: str,
) -> TokenizerLike | None

vscode-textmate spelling of :meth:load_grammar.

load_grammar

load_grammar(
    initial_scope_name: str,
) -> TokenizerLike | None

Load a grammar and every referenced grammar before constructing it.

register_grammar

register_grammar(
    raw_grammar: IRawGrammar,
    injections: Sequence[str] | None = None,
) -> None

Register an already parsed raw grammar and optional injection scopes.

RegistryOptions


              flowchart TD
              textmate_grammar.RegistryOptions[RegistryOptions]

              

              click textmate_grammar.RegistryOptions href "" "textmate_grammar.RegistryOptions"
            

Synchronous counterpart of upstream RegistryOptions.

SyncRegistry

SyncRegistry(onig_lib: IOnigLib = DEFAULT_ONIG_LIB)

Synchronous equivalent of upstream SyncRegistry.

Methods:

get_external_grammar

get_external_grammar(
    scope_name: str,
    repository: Mapping[str, Any] | None = None,
) -> IRawGrammar | None

Look up and initialize an included grammar, cached by scope and base.

Upstream caches per concrete Grammar instance. The registry-level substitute keys by the caller's $base object as well, preventing one root grammar's base repository from leaking into another.

create_grammar

create_grammar(
    scope_name: str,
    grammar: Mapping[str, Any],
    initial_language: int = 1,
    embedded_languages: Mapping[str, int] | None = None,
    token_types: Mapping[str, int] | None = None,
    balanced_bracket_selectors: BalancedBracketSelectors
    | None = None,
    grammar_repository: Any = None,
    onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
    *,
    external_grammar: Callable[
        [str], Mapping[str, Any] | None
    ]
    | None = None,
) -> Grammar

Construct a grammar; external_grammar is the registry integration seam.

parse_raw_grammar

parse_raw_grammar(
    content: str, file_path: str | None = None
) -> Any

Parse a grammar from content (parseRawGrammar.ts upstream).

Routing is by file path only: paths ending in .json parse as JSON, everything else (including .jsonc, .yaml and .xml) parses as plist — upstream supports nothing else.

engine

Public vscode-textmate-shaped tokenization API.

Modules:

  • debug –

    Debug flags ported from vscode-textmate's src/debug.ts.

  • matcher –

    Port of vscode-textmate's src/matcher.ts.

  • oniglib –

    Port of vscode-textmate's src/onigLib.ts — the oniguruma library adapter.

  • plist –

    Faithful port of microsoft/vscode-textmate src/plist.ts (v9.3.2).

  • raw_grammar –

    Faithful port of microsoft/vscode-textmate src/rawGrammar.ts,

  • regexsource –

    Port of the RegExpSource machinery from vscode-textmate's src/rule.ts.

  • registry –

    Grammar registration and dependency loading from vscode-textmate v9.3.2.

  • rules –

    Port of the Rule classes, RuleRegistry and the rule factory from

  • tokenize –

    The vscode-textmate v9.3.2 state model and line tokenization engine.

Classes:

  • FindOption –

    Port of the FindOption const enum from onigLib.ts (onigLib.ts:23-41).

  • Grammar –

    A loaded grammar and its vscode-textmate line tokenizer.

  • ILocation –

    Source location of a grammar construct (ILocation upstream).

  • IOnigCaptureIndex –

    Port of IOnigCaptureIndex (onigLib.ts:12-16).

  • IOnigLib –

    Port of the IOnigLib interface (onigLib.ts:7-10): the library surface

  • IOnigMatch –

    Port of IOnigMatch (onigLib.ts:18-21): the index of the matching pattern

  • OnigLib –

    The concrete onigurumacffi-backed :class:IOnigLib — the role

  • OnigScanner –

    Port of OnigScanner from vscode-oniguruma — the scanner behind

  • OnigString –

    Port of OnigString (onigLib.ts:48-51).

  • Registry –

    Registry options, dependency resolution, and lazy grammar construction.

  • RegistryOptions –

    Synchronous counterpart of upstream RegistryOptions.

  • StateStackImpl –

    Port of grammar.ts StateStackImpl (including line-scoped positions).

  • SyncRegistry –

    Synchronous equivalent of upstream SyncRegistry.

Functions:

  • create_grammar –

    Construct a grammar; external_grammar is the registry integration seam.

  • disposeOnigString –

    Port of disposeOnigString (onigLib.ts:53-57).

  • parse_raw_grammar –

    Parse a grammar from content (parseRawGrammar.ts upstream).

FindOption


              flowchart TD
              textmate_grammar.engine.FindOption[FindOption]

              

              click textmate_grammar.engine.FindOption href "" "textmate_grammar.engine.FindOption"
            

Port of the FindOption const enum from onigLib.ts (onigLib.ts:23-41).

The bit values are vscode-textmate's own numbering; they are translated to the oniguruma search options named in the upstream member comments (:func:_translate_options). DEBUG_CALL is a debugging marker only and is dropped, like the debugCall flag is in vscode-oniguruma's non-debug path.

Grammar

Grammar(
    scope_name: str,
    grammar: Mapping[str, Any],
    initial_language: int = 1,
    embedded_languages: Mapping[str, int] | None = None,
    token_types: Mapping[str, int] | None = None,
    balanced_bracket_selectors: BalancedBracketSelectors
    | None = None,
    grammar_repository: Any = None,
    onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
    *,
    external_grammar: Callable[
        [str], Mapping[str, Any] | None
    ]
    | None = None,
)

              flowchart TD
              textmate_grammar.engine.Grammar[Grammar]
              textmate_grammar.engine.rules.RuleRegistry[RuleRegistry]
              textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
              textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
              textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]

                              textmate_grammar.engine.rules.RuleRegistry --> textmate_grammar.engine.Grammar
                                textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.rules.RuleRegistry
                                textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                
                textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                


                textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.Grammar
                                textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                
                textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                



              click textmate_grammar.engine.Grammar href "" "textmate_grammar.engine.Grammar"
              click textmate_grammar.engine.rules.RuleRegistry href "" "textmate_grammar.engine.rules.RuleRegistry"
              click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
              click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
              click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
            

A loaded grammar and its vscode-textmate line tokenizer.

Methods:

  • create_onig_scanner –

    Port of Grammar.createOnigScanner (grammar.ts:177-179).

  • create_onig_string –

    Port of Grammar.createOnigString (grammar.ts:180-181).

  • get_rule –

    Port of Grammar.getRule (grammar.ts:252-254). May return the

  • register_rule –

    Port of Grammar.registerRule (grammar.ts:245-250).

create_onig_scanner

create_onig_scanner(sources: Sequence[str]) -> Any

Port of Grammar.createOnigScanner (grammar.ts:177-179).

create_onig_string

create_onig_string(string: str) -> Any

Port of Grammar.createOnigString (grammar.ts:180-181).

get_rule

get_rule(rule_id: int) -> Rule

Port of Grammar.getRule (grammar.ts:252-254). May return the (python) None of a not-yet-registered rule while its factory is still running — upstream yields undefined there, and the only reader that can act on it treats non-None like undefined (see RuleFactory.compile_patterns).

register_rule

register_rule(factory: Callable[[int], T]) -> T

Port of Grammar.registerRule (grammar.ts:245-250).

ILocation


              flowchart TD
              textmate_grammar.engine.ILocation[ILocation]

              

              click textmate_grammar.engine.ILocation href "" "textmate_grammar.engine.ILocation"
            

Source location of a grammar construct (ILocation upstream).

IOnigCaptureIndex dataclass

IOnigCaptureIndex(start: int, end: int, length: int)

Port of IOnigCaptureIndex (onigLib.ts:12-16).

IOnigLib


              flowchart TD
              textmate_grammar.engine.IOnigLib[IOnigLib]

              

              click textmate_grammar.engine.IOnigLib href "" "textmate_grammar.engine.IOnigLib"
            

Port of the IOnigLib interface (onigLib.ts:7-10): the library surface vscode-textmate expects the tokenization engine to be wired against.

Methods:

create_onig_scanner

create_onig_scanner(sources: Sequence[str]) -> OnigScanner

Port of createOnigScanner(sources).

create_onig_string

create_onig_string(string: str) -> OnigString

Port of createOnigString(str).

IOnigMatch dataclass

IOnigMatch(
    index: int,
    capture_indices: tuple[IOnigCaptureIndex, ...],
)

Port of IOnigMatch (onigLib.ts:18-21): the index of the matching pattern within the scanner and the captured regions of that match.

OnigLib

The concrete onigurumacffi-backed :class:IOnigLib — the role vscode-oniguruma plays upstream when main.ts wires up the default library.

OnigScanner

OnigScanner(sources: Sequence[str])

Port of OnigScanner from vscode-oniguruma — the scanner behind IOnigLib.createOnigScanner (onigLib.ts:43-46).

Patterns are compiled eagerly at construction, like vscode-oniguruma does; an invalid pattern raises :class:onigurumacffi.OnigError where upstream throws the oniguruma error message. Matching uses one RegSet search per call, whose ONIG_REGSET_POSITION_LEAD strategy yields the same candidate selection as vscode-oniguruma's C++ findNextOnigScannerMatch: the match beginning at the smallest offset wins and ties break toward the lowest pattern index.

Methods:

  • dispose –

    Port of the optional dispose member: drops the compiled pattern set.

  • find_best_match –

    Convenience mirroring the pre-4.0 ``IOnigScanner.findBestMatch(position,

  • find_next_match_sync –

    Port of OnigScanner.findNextMatchSync (onigLib.ts:44).

dispose

dispose() -> None

Port of the optional dispose member: drops the compiled pattern set.

find_best_match

find_best_match(
    position: int,
    line: str,
    has_anchor: bool = False,
    has_back_references: bool = False,
) -> IOnigMatch | None

Convenience mirroring the pre-4.0 IOnigScanner.findBestMatch(position, line, hasAnchor, hasBackReferences) surface named in the porting plan.

position is the scan offset, so a \G binds there: with has_anchor=True the patterns are scanned as-is, with has_anchor=False FindOption.NOT_BEGIN_POSITION makes every \G fail (the same equivalent option vscode-textmate's getFindOptions builds for allowG=False in grammar/tokenizeString.ts:549-557).

has_back_references is bookkeeping parity for the old signature only: in the current vscode-textmate, \n backreference patterns are resolved before scanning (rule.ts resolveBackReferences, ported in :mod:textmate_grammar.engine.regexsource), so the scanner has nothing to do with them — the flag is accepted and unused.

find_next_match_sync

find_next_match_sync(
    string: str | OnigString,
    start_position: int = 0,
    options: FindOption = NONE,
) -> IOnigMatch | None

Port of OnigScanner.findNextMatchSync (onigLib.ts:44).

string may be a plain string or an :class:OnigString, as upstream. start_position is the offset the scan starts at, which is where \G binds: a pattern containing \G is compiled once and still matches at whatever initial offset the caller passes.

OnigString

OnigString(content: str)

Port of OnigString (onigLib.ts:48-51).

Upstream this wraps the wasm-resident UTF-8 encoding of the string and carries the UTF-8/UTF-16 offset translation tables; Python strings are indexed by code point natively, so this is the light wrapper keeping content around, with a no-op dispose (parity of the optional dispose? member).

Methods:

  • dispose –

    No-op — there is no wasm string buffer to release.

dispose

dispose() -> None

No-op — there is no wasm string buffer to release.

Registry

Registry(
    options: RegistryOptions,
    *,
    grammar_factory: GrammarFactory | None = None,
)

Registry options, dependency resolution, and lazy grammar construction.

Methods:

  • addGrammar –

    Register and construct a grammar, matching upstream's convenience method.

  • loadGrammar –

    vscode-textmate spelling of :meth:load_grammar.

  • load_grammar –

    Load a grammar and every referenced grammar before constructing it.

  • register_grammar –

    Register an already parsed raw grammar and optional injection scopes.

addGrammar

addGrammar(
    raw_grammar: IRawGrammar,
    injections: Sequence[str] = (),
    initial_language: int = 0,
    embedded_languages: Mapping[str, int] | None = None,
) -> TokenizerLike

Register and construct a grammar, matching upstream's convenience method.

loadGrammar

loadGrammar(
    initial_scope_name: str,
) -> TokenizerLike | None

vscode-textmate spelling of :meth:load_grammar.

load_grammar

load_grammar(
    initial_scope_name: str,
) -> TokenizerLike | None

Load a grammar and every referenced grammar before constructing it.

register_grammar

register_grammar(
    raw_grammar: IRawGrammar,
    injections: Sequence[str] | None = None,
) -> None

Register an already parsed raw grammar and optional injection scopes.

RegistryOptions


              flowchart TD
              textmate_grammar.engine.RegistryOptions[RegistryOptions]

              

              click textmate_grammar.engine.RegistryOptions href "" "textmate_grammar.engine.RegistryOptions"
            

Synchronous counterpart of upstream RegistryOptions.

StateStackImpl

StateStackImpl(
    parent: StateStackImpl | None,
    rule_id: int,
    enter_pos: int,
    anchor_pos: int,
    begin_rule_captured_eol: bool,
    end_rule: str | None,
    name_scopes_list: AttributedScopeStack | None,
    content_name_scopes_list: AttributedScopeStack | None,
)

Port of grammar.ts StateStackImpl (including line-scoped positions).

SyncRegistry

SyncRegistry(onig_lib: IOnigLib = DEFAULT_ONIG_LIB)

Synchronous equivalent of upstream SyncRegistry.

Methods:

get_external_grammar

get_external_grammar(
    scope_name: str,
    repository: Mapping[str, Any] | None = None,
) -> IRawGrammar | None

Look up and initialize an included grammar, cached by scope and base.

Upstream caches per concrete Grammar instance. The registry-level substitute keys by the caller's $base object as well, preventing one root grammar's base repository from leaking into another.

create_grammar

create_grammar(
    scope_name: str,
    grammar: Mapping[str, Any],
    initial_language: int = 1,
    embedded_languages: Mapping[str, int] | None = None,
    token_types: Mapping[str, int] | None = None,
    balanced_bracket_selectors: BalancedBracketSelectors
    | None = None,
    grammar_repository: Any = None,
    onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
    *,
    external_grammar: Callable[
        [str], Mapping[str, Any] | None
    ]
    | None = None,
) -> Grammar

Construct a grammar; external_grammar is the registry integration seam.

disposeOnigString

disposeOnigString(string: OnigString) -> None

Port of disposeOnigString (onigLib.ts:53-57).

parse_raw_grammar

parse_raw_grammar(
    content: str, file_path: str | None = None
) -> Any

Parse a grammar from content (parseRawGrammar.ts upstream).

Routing is by file path only: paths ending in .json parse as JSON, everything else (including .jsonc, .yaml and .xml) parses as plist — upstream supports nothing else.

debug

Debug flags ported from vscode-textmate's src/debug.ts.

Classes:

  • DebugFlags –

    Flags controlling optional debug behavior.

DebugFlags

Flags controlling optional debug behavior.

Methods:

  • is_in_debug_mode –

    Read the flag at a module reload boundary, as the parser does.

is_in_debug_mode staticmethod
is_in_debug_mode() -> bool

Read the flag at a module reload boundary, as the parser does.

matcher

Port of vscode-textmate's src/matcher.ts.

Scope selectors such as text.html.php - (meta.embedded | meta.tag), L:text.html.php meta.tag are parsed into small composable predicate functions evaluated against a scope path. The semantics are intentionally 1:1 with the TypeScript source; comments throughout this module reference the corresponding sections of matcher.ts (and of grammar.ts/theme.ts where matcher.ts delegates the name-matching logic).

Note on ordering: the only ordering information matcher.ts itself produces is the L: (priority -1) / R: (priority 1) selector prefix, surfaced as :attr:MatcherWithPriority.priority. Sorting injections by that priority happens in grammar.ts (task 3b); specificity ordering of theme rules lives in theme.ts and is out of this plan's scope.

Classes:

  • MatcherWithPriority –

    matcher.ts interface MatcherWithPriority<T> { matcher; priority }.

  • ScopePath –

    The parent-linked scope chain matchers are evaluated against.

Functions:

MatcherWithPriority


              flowchart TD
              textmate_grammar.engine.matcher.MatcherWithPriority[MatcherWithPriority]

              

              click textmate_grammar.engine.matcher.MatcherWithPriority href "" "textmate_grammar.engine.matcher.MatcherWithPriority"
            

matcher.ts interface MatcherWithPriority<T> { matcher; priority }.

matcher is a predicate over whatever input the caller evaluates against (grammar.ts uses the flat scope-name list of the current token); priority is the parsed L:/R: prefix.

ScopePath


              flowchart TD
              textmate_grammar.engine.matcher.ScopePath[ScopePath]

              

              click textmate_grammar.engine.matcher.ScopePath href "" "textmate_grammar.engine.matcher.ScopePath"
            

The parent-linked scope chain matchers are evaluated against.

A scope path is a linked chain of scope names: the node a caller holds is the innermost scope, .name its scope name and .parent points towards the shallower scope; the root scope has parent = None. Materialized root-first this equals theme.ts' ScopeStack.getSegments() (i.e. grammar.ts' getScopeNames()). Minimal on purpose; the state stack built in task 3a satisfies this protocol structurally.

create_matchers

create_matchers(
    selector: str, matches_name: MatchesName[T]
) -> list[MatcherWithPriority]

Port of matcher.ts createMatchers.

Parses selector into its comma-separated branches and returns one MatcherWithPriority per branch. Callers treat the branch matchers as alternatives (matcher.ts consumers use matchers.some(...)). Only priority carries ordering information; the matchers are pure predicates.

match_scope_identifiers

match_scope_identifiers(
    identifiers: list[str], scopes: list[str]
) -> bool

Port of grammar.ts nameMatcher, the matches_name matcher.ts is fed.

Every identifier must occur as a subsequence of scopes, scanning strictly left to right: identifier order matters and each identifier consumes its position (repeated identifiers need repeated scopes). A stack scope matches an identifier via :func:scopes_are_matching.

match_scope_names

match_scope_names(
    identifiers: list[str], scopes_path: ScopePath | None
) -> bool

Name matcher over a :class:ScopePath parent chain.

Equivalent to grammar.ts nameMatcher fed getScopeNames(): the chain is collected from the innermost node through the parents and then reversed to root-first order (theme.ts ScopeStack.getSegments()).

scopes_are_matching

scopes_are_matching(
    scope_name: str, selector_scope: str
) -> bool

Port of grammar.ts scopesAreMatching (matcher.ts delegates to it).

True when the stack scope scope_name equals selector_scope or extends it at a . boundary: text.html matches text.html.php but not text.htmlx.

oniglib

Port of vscode-textmate's src/onigLib.ts — the oniguruma library adapter.

vscode-textmate drives tokenization through an injectable IOnigLib that provides scanners and strings. Upstream this is backed by vscode-oniguruma (a wasm build of oniguruma ~6.9). Here the same surface is provided on top of :mod:onigurumacffi, an ffi binding of oniguruma 6.9 — the same engine family, so the semantics carry over:

  • Pattern selection (leftmost match wins, ties resolved by the lowest pattern index) is done by oniguruma's RegSet API (onig_regset_search with ONIG_REGSET_POSITION_LEAD), which is exactly the strategy vscode-oniguruma's C++ findNextOnigScannerMatch uses for short strings.

  • The critical semantic (see the parity plan): \G matches exactly at the start offset of the scan, i.e. at the start_position passed to each :meth:OnigScanner.find_next_match_sync call, not at a stored anchor. oniguruma binds \G to the search start, and vscode-textmate passes the token offset per call — Pattern.search(content, start=start_position) reproduces this 1:1. A pattern compiled once therefore matches at different offsets for different start_position values.

Capture indices are returned in Python code-point offsets (consistent with the rest of the Python port; upstream returns UTF-16 offsets, having decoded the UTF-8 wasm buffer). Unmatched capture groups — which oniguruma reports with offsets of -1 — are mapped to start == end == 0, the same clamp vscode-oniguruma applies when its convertUtf8OffsetToUtf16 encounters values below zero.

Every non-obvious piece references the ported section of vscode-oniguruma / onigLib.ts.

Classes:

  • FindOption –

    Port of the FindOption const enum from onigLib.ts (onigLib.ts:23-41).

  • IOnigCaptureIndex –

    Port of IOnigCaptureIndex (onigLib.ts:12-16).

  • IOnigLib –

    Port of the IOnigLib interface (onigLib.ts:7-10): the library surface

  • IOnigMatch –

    Port of IOnigMatch (onigLib.ts:18-21): the index of the matching pattern

  • OnigLib –

    The concrete onigurumacffi-backed :class:IOnigLib — the role

  • OnigScanner –

    Port of OnigScanner from vscode-oniguruma — the scanner behind

  • OnigString –

    Port of OnigString (onigLib.ts:48-51).

Functions:

FindOption


              flowchart TD
              textmate_grammar.engine.oniglib.FindOption[FindOption]

              

              click textmate_grammar.engine.oniglib.FindOption href "" "textmate_grammar.engine.oniglib.FindOption"
            

Port of the FindOption const enum from onigLib.ts (onigLib.ts:23-41).

The bit values are vscode-textmate's own numbering; they are translated to the oniguruma search options named in the upstream member comments (:func:_translate_options). DEBUG_CALL is a debugging marker only and is dropped, like the debugCall flag is in vscode-oniguruma's non-debug path.

IOnigCaptureIndex dataclass

IOnigCaptureIndex(start: int, end: int, length: int)

Port of IOnigCaptureIndex (onigLib.ts:12-16).

IOnigLib


              flowchart TD
              textmate_grammar.engine.oniglib.IOnigLib[IOnigLib]

              

              click textmate_grammar.engine.oniglib.IOnigLib href "" "textmate_grammar.engine.oniglib.IOnigLib"
            

Port of the IOnigLib interface (onigLib.ts:7-10): the library surface vscode-textmate expects the tokenization engine to be wired against.

Methods:

create_onig_scanner
create_onig_scanner(sources: Sequence[str]) -> OnigScanner

Port of createOnigScanner(sources).

create_onig_string
create_onig_string(string: str) -> OnigString

Port of createOnigString(str).

IOnigMatch dataclass

IOnigMatch(
    index: int,
    capture_indices: tuple[IOnigCaptureIndex, ...],
)

Port of IOnigMatch (onigLib.ts:18-21): the index of the matching pattern within the scanner and the captured regions of that match.

OnigLib

The concrete onigurumacffi-backed :class:IOnigLib — the role vscode-oniguruma plays upstream when main.ts wires up the default library.

OnigScanner

OnigScanner(sources: Sequence[str])

Port of OnigScanner from vscode-oniguruma — the scanner behind IOnigLib.createOnigScanner (onigLib.ts:43-46).

Patterns are compiled eagerly at construction, like vscode-oniguruma does; an invalid pattern raises :class:onigurumacffi.OnigError where upstream throws the oniguruma error message. Matching uses one RegSet search per call, whose ONIG_REGSET_POSITION_LEAD strategy yields the same candidate selection as vscode-oniguruma's C++ findNextOnigScannerMatch: the match beginning at the smallest offset wins and ties break toward the lowest pattern index.

Methods:

  • dispose –

    Port of the optional dispose member: drops the compiled pattern set.

  • find_best_match –

    Convenience mirroring the pre-4.0 ``IOnigScanner.findBestMatch(position,

  • find_next_match_sync –

    Port of OnigScanner.findNextMatchSync (onigLib.ts:44).

dispose
dispose() -> None

Port of the optional dispose member: drops the compiled pattern set.

find_best_match
find_best_match(
    position: int,
    line: str,
    has_anchor: bool = False,
    has_back_references: bool = False,
) -> IOnigMatch | None

Convenience mirroring the pre-4.0 IOnigScanner.findBestMatch(position, line, hasAnchor, hasBackReferences) surface named in the porting plan.

position is the scan offset, so a \G binds there: with has_anchor=True the patterns are scanned as-is, with has_anchor=False FindOption.NOT_BEGIN_POSITION makes every \G fail (the same equivalent option vscode-textmate's getFindOptions builds for allowG=False in grammar/tokenizeString.ts:549-557).

has_back_references is bookkeeping parity for the old signature only: in the current vscode-textmate, \n backreference patterns are resolved before scanning (rule.ts resolveBackReferences, ported in :mod:textmate_grammar.engine.regexsource), so the scanner has nothing to do with them — the flag is accepted and unused.

find_next_match_sync
find_next_match_sync(
    string: str | OnigString,
    start_position: int = 0,
    options: FindOption = NONE,
) -> IOnigMatch | None

Port of OnigScanner.findNextMatchSync (onigLib.ts:44).

string may be a plain string or an :class:OnigString, as upstream. start_position is the offset the scan starts at, which is where \G binds: a pattern containing \G is compiled once and still matches at whatever initial offset the caller passes.

OnigString

OnigString(content: str)

Port of OnigString (onigLib.ts:48-51).

Upstream this wraps the wasm-resident UTF-8 encoding of the string and carries the UTF-8/UTF-16 offset translation tables; Python strings are indexed by code point natively, so this is the light wrapper keeping content around, with a no-op dispose (parity of the optional dispose? member).

Methods:

  • dispose –

    No-op — there is no wasm string buffer to release.

dispose
dispose() -> None

No-op — there is no wasm string buffer to release.

disposeOnigString

disposeOnigString(string: OnigString) -> None

Port of disposeOnigString (onigLib.ts:53-57).

plist

Faithful port of microsoft/vscode-textmate src/plist.ts (v9.3.2).

A very fast dependency-free property-list (XML) parser for TextMate grammars. It is deliberately not a general XML parser: it understands exactly the tags TextMate grammars use (dict, array, key, string, real, integer, date, data, true, false, plist) and skips processing instructions, comments and doctypes. Semantics shipped here match the TypeScript implementation 1:1, including its errors and quirks:

  • Quirks preserved: closing tags and the plist tag are matched by string prefix (</dictionary> closes as </dict>); unterminated values at EOF are accepted; there is no CDATA support (<![CDATA[...]]> is skipped to the next > like a comment); named entities only cover &amp;&lt;&gt;&quot;&apos;, while numeric entities are decimal &#NNN; and lowercase-hex &#xNNN; (uppercase hex digits are not decoded; &#x4A; stays literal); <true>/<false> ignore their text content; <integer> values always yield a JS number (a Python float).
  • Locations ($vscodeTextmateLocation entries) are only produced in the parse_with_location debug path, exactly like upstream.
  • Errors are ValueError with the upstream message format::

    Near offset : ~~~<50 chars of input>~~~

See plist.ts in microsoft/vscode-textmate for the reference implementation.

Functions:

  • parse_plist –

    A very fast plist parser (plist.parsePLIST upstream).

  • parse_with_location –

    Parse content, optionally embedding locations (debug mode).

parse_plist

parse_plist(content: str) -> Any

A very fast plist parser (plist.parsePLIST upstream).

parse_with_location

parse_with_location(
    content: str,
    filename: str | None,
    location_key_name: str | None,
) -> Any

Parse content, optionally embedding locations (debug mode).

raw_grammar

Faithful port of microsoft/vscode-textmate src/rawGrammar.ts, src/parseRawGrammar.ts and src/json.ts (v9.3.2).

rawGrammar.ts in the pinned upstream release is purely type-level (IRawGrammar, IRawRule, IRawCaptures, IRawRepository, ILocation): there is no runtime RawGrammar/RawCaptures class and no valueAsString coercion in the pinned source (those notions date back to pre-1.0 vscode-textmate; the modern engine simply runs on the any objects produced by JSON.parse/plist.parsePLIST, so upstream types are runtime lies). The upstream interfaces are mirrored here as TypedDict shapes; the scalar coercion helper :func:value_as_string covers the one coercion downstream rule-building code must not guess at (JS String() semantics for string-typed fields that may technically hold plist scalars).

parseRawGrammar.ts semantics, exactly as upstream:

  • a file path ending with .json (case sensitive suffix test: /\.json$/) → JSON grammar; everything else → plist grammar. There is no content sniffing and no YAML support upstream: unknown extensions are handed to the plist parser, which fails with Near offset ...: expected < for non-XML content (preserved here).
  • the production path is strict JSON: upstream calls JSON.parse and only uses the hand-rolled parseJSON tokenizer from json.ts in debug mode. json.loads mirrors JSON.parse (strict JSON: no comments, no trailing commas).

json.ts is ported in full and exported as :func:parse_json for the debug path (VSCODE_TEXTMATE_DEBUG environment variable, standing in for upstream DebugFlags.InDebugMode). Its tolerances — verified against the TS source — are not jsonc: it supports neither comments nor trailing commas; its leniency lies in JS parseFloat-style number tokens (1-2 parses as 1) and in accepting trailing garbage after the root construct.

Classes:

  • ILocation –

    Source location of a grammar construct (ILocation upstream).

  • JSONStreamState –

    Tokenizer state (JSONStreamState upstream).

Functions:

  • parse_json –

    parseJSON upstream — the hand-rolled, JS-quirky JSON tokenizer.

  • parse_raw_grammar –

    Parse a grammar from content (parseRawGrammar.ts upstream).

  • value_as_string –

    Coerce a raw grammar scalar to a string with JS String() semantics.

ILocation


              flowchart TD
              textmate_grammar.engine.raw_grammar.ILocation[ILocation]

              

              click textmate_grammar.engine.raw_grammar.ILocation href "" "textmate_grammar.engine.raw_grammar.ILocation"
            

Source location of a grammar construct (ILocation upstream).

JSONStreamState

JSONStreamState(source: str)

Tokenizer state (JSONStreamState upstream).

next_json_token

next_json_token(
    stream_state: JSONStreamState, out_token: _JSONToken
) -> bool

Produce the next token (nextJSONToken upstream).

Precondition upstream: the source is known to be valid JSON. Note the quirks: on False (EOS or an invalid keyword tail) the stream state is not updated; a failing string/keyword read leaves out_token with the type set but no value/updated state.

parse_json

parse_json(
    source: str, filename: str | None, with_metadata: bool
) -> Any

parseJSON upstream — the hand-rolled, JS-quirky JSON tokenizer.

Only used on the debug path upstream (VSCODE_TEXTMATE_DEBUG); embeds $vscodeTextmateLocation when with_metadata is set.

Like the TS engine, positions are indexed by UTF-16 code units; python values are returned with surrogate pairs recombined into astral characters (see plist._units_of/_text_of).

parse_raw_grammar

parse_raw_grammar(
    content: str, file_path: str | None = None
) -> Any

Parse a grammar from content (parseRawGrammar.ts upstream).

Routing is by file path only: paths ending in .json parse as JSON, everything else (including .jsonc, .yaml and .xml) parses as plist — upstream supports nothing else.

value_as_string

value_as_string(value: Any) -> str

Coerce a raw grammar scalar to a string with JS String() semantics.

Practical fallback for rule-building code that must not guess how grammar scalars (which may technically be <integer>/<real>/<true> values in wayward tmLanguage files) render as strings:

  • true/false → "true"/"false" (JS, not Python capitalization)
  • numbers → shortest double representation (1.0 → "1"), NaN/Infinity match JS tokens
  • everything else → str(value)

regexsource

Port of the RegExpSource machinery from vscode-textmate's src/rule.ts.

This module ports from rule.ts (version 9.3.2, the pinned reference clone): IRegExpSourceAnchorCache (rule.ts:575-580), RegExpSource (rule.ts:582-735), RegExpSourceList (rule.ts:744-856) and CompiledRule (rule.ts:858-894), plus the two regex-source helpers they depend on from src/utils.ts: escapeRegExpCharacters (utils.ts:159-161) and the RegexSource class (utils.ts:66-115, $n/${n:/command} capture expansion used for scope names).

Semantics are 1:1 with the reference; naming is pythonic and every non-obvious piece carries a comment referencing the ported lines of rule.ts / utils.ts.

Classes:

  • AnchorCache –

    Port of IRegExpSourceAnchorCache (rule.ts:575-580): the four resolved

  • CompiledRule –

    Port of rule.ts's CompiledRule (rule.ts:858-894): a set of pattern

  • IFindNextMatchResult –

    Port of IFindNextMatchResult (rule.ts:896-899): the matched pattern's

  • RegExpSource –

    Port of rule.ts's RegExpSource (rule.ts:582-735): a pattern source from

  • RegExpSourceList –

    Port of rule.ts's RegExpSourceList (rule.ts:744-856): a growing list of

  • RegexSource –

    Port of utils.ts's static RegexSource class (utils.ts:66-115): expands

Functions:

AnchorCache dataclass

AnchorCache(a0_g0: str, a0_g1: str, a1_g0: str, a1_g1: str)

Port of IRegExpSourceAnchorCache (rule.ts:575-580): the four resolved variants of a source with anchors, named after which anchors remain allowed. A1 = \A kept, G1 = \G kept; disallowed anchors were rewritten to the U+FFFF sentinel, which can virtually never occur in scanned text.

CompiledRule

CompiledRule(
    onig_lib: IOnigLib,
    reg_exps: Sequence[str],
    rules: Sequence[int],
)

Port of rule.ts's CompiledRule (rule.ts:858-894): a set of pattern sources compiled once into an :class:OnigScanner, resolving matches back to the rule ids the patterns belong to.

Methods:

  • dispose –

    Port of CompiledRule.dispose (rule.ts:865-869).

  • find_next_match_sync –

    Port of CompiledRule.findNextMatchSync (rule.ts:879-893): scans and

dispose
dispose() -> None

Port of CompiledRule.dispose (rule.ts:865-869).

find_next_match_sync
find_next_match_sync(
    string: str | OnigString,
    start_position: int,
    options: FindOption = NONE,
) -> IFindNextMatchResult | None

Port of CompiledRule.findNextMatchSync (rule.ts:879-893): scans and maps the matched pattern index back to IFindNextMatchResult.

IFindNextMatchResult dataclass

IFindNextMatchResult(
    rule_id: int,
    capture_indices: tuple[IOnigCaptureIndex, ...],
)

Port of IFindNextMatchResult (rule.ts:896-899): the matched pattern's rule id and its capture indices.

RegExpSource

RegExpSource(reg_exp_source: str, rule_id: int)

Port of rule.ts's RegExpSource (rule.ts:582-735): a pattern source from the grammar, preprocessed on construction (\z workaround), detecting whether it uses anchors (\A/\G) or \n backreferences and able to resolve the anchor variants via its anchor cache.

Walks the source once: \z (oniguruma-only, kept in tmlanguage sources) is rewritten to a JS-compatible equivalent and occurrences of \A/\G mark the source as anchored. Note that escaped anchors inside character classes are flagged too — upstream does the same (rule.ts:600-613).

Methods:

  • clone –

    Port of RegExpSource.clone (rule.ts:641-643).

  • resolve_anchors –

    Port of RegExpSource.resolveAnchors (rule.ts:716-734): the source

  • resolve_back_references –

    Port of RegExpSource.resolveBackReferences (rule.ts:656-664):

  • set_source –

    Port of RegExpSource.setSource (rule.ts:645-654): rebuilds the anchor

clone
clone() -> RegExpSource

Port of RegExpSource.clone (rule.ts:641-643).

resolve_anchors
resolve_anchors(allow_a: bool, allow_g: bool) -> str

Port of RegExpSource.resolveAnchors (rule.ts:716-734): the source variant with the disallowed anchors neutralized; non-anchored sources are returned verbatim (rule.ts:717-719).

resolve_back_references
resolve_back_references(
    line_text: str,
    capture_indices: Sequence[IOnigCaptureIndex],
) -> str

Port of RegExpSource.resolveBackReferences (rule.ts:656-664): substitutes \n-style backreferences in the source with the regexp-escaped content of capture n from the (begin) match, e.g. for end patterns echoing the begun content. Unmatched or out-of-range captures substitute '' — upstream capturedValues[...] || '' (rule.ts:662).

set_source
set_source(new_source: str) -> None

Port of RegExpSource.setSource (rule.ts:645-654): rebuilds the anchor cache from the new source — but only picks up has_anchor changes on the next construction (upstream keeps the old flag here too).

RegExpSourceList

RegExpSourceList()

Port of rule.ts's RegExpSourceList (rule.ts:744-856): a growing list of :class:RegExpSource items compiled lazily into :class:CompiledRule objects.

For lists with anchors, the compiled scanners are cached in _anchor_cache — upstream carries the four A0_G0..A1_G1 fields (rule.ts:737-742, 751-760); they are keyed here by (allow_a, allow_g).

Methods:

  • compile –

    Port of RegExpSourceList.compile (rule.ts:812-818): compiles the

  • compile_ag –

    Port of RegExpSourceList.compileAG (rule.ts:820-850): non-anchored

  • push –

    Port of RegExpSourceList.push (rule.ts:790-793).

  • set_source –

    Port of RegExpSourceList.setSource (rule.ts:804-810): busts the

  • unshift –

    Port of RegExpSourceList.unshift (rule.ts:795-798).

compile
compile(onig_lib: IOnigLib) -> CompiledRule

Port of RegExpSourceList.compile (rule.ts:812-818): compiles the plain sources once and caches them.

compile_ag
compile_ag(
    onig_lib: IOnigLib, allow_a: bool, allow_g: bool
) -> CompiledRule

Port of RegExpSourceList.compileAG (rule.ts:820-850): non-anchored lists share the plain compile (rule.ts:821-822); anchored lists get one cached compiled variant per allowed anchor combination.

push
push(item: RegExpSource) -> None

Port of RegExpSourceList.push (rule.ts:790-793).

set_source
set_source(index: int, new_source: str) -> None

Port of RegExpSourceList.setSource (rule.ts:804-810): busts the caches when the source of an item actually changes.

unshift
unshift(item: RegExpSource) -> None

Port of RegExpSourceList.unshift (rule.ts:795-798).

RegexSource

Port of utils.ts's static RegexSource class (utils.ts:66-115): expands $n and ${n:/downcase|upcase} captures in scope names against the text a pattern captured.

Methods:

  • hasCaptures –

    Port of RegexSource.hasCaptures (utils.ts:68-75).

  • replaceCaptures –

    Port of RegexSource.replaceCaptures (utils.ts:77-115): replaces each

hasCaptures staticmethod
hasCaptures(regex_source: str | None) -> bool

Port of RegexSource.hasCaptures (utils.ts:68-75).

replaceCaptures staticmethod
replaceCaptures(
    regex_source: str,
    capture_source: str,
    capture_indices: Sequence[IOnigCaptureIndex],
) -> str

Port of RegexSource.replaceCaptures (utils.ts:77-115): replaces each $n/${n:/command} occurrence with the content capture n covers in capture_source, applying the optional case command. References to captures that don't exist are left verbatim. Unmatched captures (oniguruma -1 offsets) contribute '' via the empty substring slice.

escapeRegExpCharacters

escapeRegExpCharacters(value: str) -> str

Port of utils.ts escapeRegExpCharacters (utils.ts:159-161) — escapes the characters that are special in regular expressions in the given string.

registry

Grammar registration and dependency loading from vscode-textmate v9.3.2.

The upstream Registry and SyncRegistry create the concrete Grammar class. That class is not available until task 3b, so the one deliberate seam here is an injectable grammar_factory(location, raw_grammar). Its default raises NotImplementedError; the later tokenizer task supplies the real constructor.

Upstream exposes asynchronous methods because its loader returns promises. This port intentionally uses synchronous callables returning raw grammar mappings (or None); dependency traversal, load caching, registration, and construction otherwise retain the upstream behavior. onig_lib is carried in the options for the eventual Grammar wiring, but is not consumed by this seam yet.

Classes:

Functions:

ExternalReferenceCollector

ExternalReferenceCollector()

Deduplicating collector; raw-rule visitation uses identity like JS Set.

Registry

Registry(
    options: RegistryOptions,
    *,
    grammar_factory: GrammarFactory | None = None,
)

Registry options, dependency resolution, and lazy grammar construction.

Methods:

  • addGrammar –

    Register and construct a grammar, matching upstream's convenience method.

  • loadGrammar –

    vscode-textmate spelling of :meth:load_grammar.

  • load_grammar –

    Load a grammar and every referenced grammar before constructing it.

  • register_grammar –

    Register an already parsed raw grammar and optional injection scopes.

addGrammar
addGrammar(
    raw_grammar: IRawGrammar,
    injections: Sequence[str] = (),
    initial_language: int = 0,
    embedded_languages: Mapping[str, int] | None = None,
) -> TokenizerLike

Register and construct a grammar, matching upstream's convenience method.

loadGrammar
loadGrammar(
    initial_scope_name: str,
) -> TokenizerLike | None

vscode-textmate spelling of :meth:load_grammar.

load_grammar
load_grammar(
    initial_scope_name: str,
) -> TokenizerLike | None

Load a grammar and every referenced grammar before constructing it.

register_grammar
register_grammar(
    raw_grammar: IRawGrammar,
    injections: Sequence[str] | None = None,
) -> None

Register an already parsed raw grammar and optional injection scopes.

RegistryOptions


              flowchart TD
              textmate_grammar.engine.registry.RegistryOptions[RegistryOptions]

              

              click textmate_grammar.engine.registry.RegistryOptions href "" "textmate_grammar.engine.registry.RegistryOptions"
            

Synchronous counterpart of upstream RegistryOptions.

ScopeDependencyProcessor

ScopeDependencyProcessor(
    repo: SyncRegistry, initial_scope_name: str
)

Queue-based port of ScopeDependencyProcessor (lines 59-103).

SyncRegistry

SyncRegistry(onig_lib: IOnigLib = DEFAULT_ONIG_LIB)

Synchronous equivalent of upstream SyncRegistry.

Methods:

get_external_grammar
get_external_grammar(
    scope_name: str,
    repository: Mapping[str, Any] | None = None,
) -> IRawGrammar | None

Look up and initialize an included grammar, cached by scope and base.

Upstream caches per concrete Grammar instance. The registry-level substitute keys by the caller's $base object as well, preventing one root grammar's base repository from leaking into another.

TokenizerLike


              flowchart TD
              textmate_grammar.engine.registry.TokenizerLike[TokenizerLike]

              

              click textmate_grammar.engine.registry.TokenizerLike href "" "textmate_grammar.engine.registry.TokenizerLike"
            

Minimal interface of a constructed grammar owned by the registry.

Methods:

  • dispose –

    Release tokenizer resources, when present.

dispose
dispose() -> None

Release tokenizer resources, when present.

TopLevelRepositoryRuleReference dataclass

TopLevelRepositoryRuleReference(
    scope_name: str, rule_name: str
)

Reference to one named rule in a grammar's top-level repository.

TopLevelRuleReference dataclass

TopLevelRuleReference(scope_name: str)

Reference to an entire grammar, as in grammarDependencies.ts:15-22.

collect_includes

collect_includes(
    grammar: IRawGrammar,
) -> list[AbsoluteRuleReference]

Collect external references from a grammar's own rules and injections.

The result is an empty-base traversal useful for inspection. Full dependency loading should use :func:get_grammar_dependencies, which resolves local $base and scope references against the registry like upstream does.

get_grammar_dependencies

get_grammar_dependencies(
    registry: SyncRegistry, scope_name: str
) -> list[str]

Return all reachable full grammar scopes, including injection scopes.

rules

Port of the Rule classes, RuleRegistry and the rule factory from vscode-textmate's src/rule.ts (version 9.3.2, the pinned reference clone).

Ported from rule.ts: endRuleId/whileRuleId (rule.ts:17-20), the Rule base class (rule.ts:43-89), CaptureRule (rule.ts:96-120), MatchRule (rule.ts:122-164), IncludeOnlyRule (rule.ts:166-207), BeginEndRule (rule.ts:209-288), BeginWhileRule (rule.ts:290-379) and the RuleFactory (rule.ts:381-573) — i.e. createCaptureRule, getCompiledRuleId, _compileCaptures and _compilePatterns. The registry lives in Grammar upstream: registerRule/getRule are grammar.ts:245-254; the class :class:RuleRegistry here extracts exactly that behaviour.

parse_include ports parseInclude and IncludeReferenceKind from src/grammar/grammarDependencies.ts (grammarDependencies.ts:247-295) — the minimal piece the rule factory needs for include resolution. init_grammar ports initGrammar (grammar.ts:410-421), which installs the $self/$base repository hooks.

Semantics are 1:1 with the reference; naming is pythonic where it can be (upstream member names are kept when the python port is simply "same words"), and every non-obvious piece carries a comment referencing the ported lines of rule.ts / grammar.ts / grammarDependencies.ts.

Classes:

  • BeginEndRule –

    Port of BeginEndRule (rule.ts:209-288): begin/end with captures

  • BeginWhileRule –

    Port of BeginWhileRule (rule.ts:290-379): begin plus a while

  • CaptureRule –

    Port of CaptureRule (rule.ts:96-120) — one entry of a captures map.

  • CompilePatternsResult –

    Port of ICompilePatternsResult (rule.ts:91-94): the ordered rule ids of

  • IRuleFactoryHelper –

    Port of IRuleFactoryHelper (rule.ts:40-41). Upstream Grammar

  • IRuleRegistry –

    Port of IRuleRegistry (rule.ts:31-34): numeric rule registration and

  • IncludeOnlyRule –

    Port of IncludeOnlyRule (rule.ts:166-207): a rule made purely of

  • IncludeReference –

    Port of the IncludeReference union (grammarDependencies.ts:240-276);

  • IncludeReferenceKind –

    Port of IncludeReferenceKind (grammarDependencies.ts:247-253).

  • MatchRule –

    Port of MatchRule (rule.ts:122-164): a single match pattern with

  • Rule –

    Port of the abstract Rule (rule.ts:43-89): identity, name/

  • RuleFactory –

    Port of RuleFactory (rule.ts:381-573).

  • RuleRegistry –

    Port of the rule-id <-> Rule bookkeeping that lives in Grammar

Functions:

  • create_rule –

    Mirror of RuleFactory.getCompiledRuleId (task-card spelling); prefer

  • init_grammar –

    Port of initGrammar (grammar.ts:410-421): deep-copies the raw grammar

  • parse_include –

    Port of parseInclude (grammarDependencies.ts:278-295): $base /

BeginEndRule

BeginEndRule(
    location: ILocation,
    id: int,
    name: str | None,
    contentName: str | None,
    begin: str,
    beginCaptures: list[CaptureRule | None],
    end: str | None,
    endCaptures: list[CaptureRule | None],
    applyEndPatternLast: bool | None,
    patterns: CompilePatternsResult,
)

              flowchart TD
              textmate_grammar.engine.rules.BeginEndRule[BeginEndRule]
              textmate_grammar.engine.rules.Rule[Rule]

                              textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.BeginEndRule
                


              click textmate_grammar.engine.rules.BeginEndRule href "" "textmate_grammar.engine.rules.BeginEndRule"
              click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
            

Port of BeginEndRule (rule.ts:209-288): begin/end with captures on both, contained patterns and applyEndPatternLast placement.

Methods:

Attributes:

debugBeginRegExp property
debugBeginRegExp: str

Port of debugBeginRegExp (rule.ts:240-242).

debugEndRegExp property
debugEndRegExp: str

Port of debugEndRegExp (rule.ts:244-246).

debugName property
debugName: str

Port of Rule.debugName (rule.ts:65-68).

getContentName
getContentName(
    lineText: str,
    captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None

Port of Rule.getContentName (rule.ts:77-82).

getEndWithResolvedBackReferences
getEndWithResolvedBackReferences(
    lineText: str,
    captureIndices: Sequence[IOnigCaptureIndex],
) -> str

Port of getEndWithResolvedBackReferences (rule.ts:248-250): the end pattern with backreferences substituted from the begin match.

getName
getName(
    lineText: str | None,
    captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None

Port of Rule.getName (rule.ts:70-75): expands capture references in the name against the matched text; non-capturing names pass through.

BeginWhileRule

BeginWhileRule(
    location: ILocation,
    id: int,
    name: str | None,
    contentName: str | None,
    begin: str,
    beginCaptures: list[CaptureRule | None],
    while_: str,
    whileCaptures: list[CaptureRule | None],
    patterns: CompilePatternsResult,
)

              flowchart TD
              textmate_grammar.engine.rules.BeginWhileRule[BeginWhileRule]
              textmate_grammar.engine.rules.Rule[Rule]

                              textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.BeginWhileRule
                


              click textmate_grammar.engine.rules.BeginWhileRule href "" "textmate_grammar.engine.rules.BeginWhileRule"
              click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
            

Port of BeginWhileRule (rule.ts:290-379): begin plus a while condition checked at the start of every line (rule id whileRuleId = -2), with the contained patterns compiled as for BeginEndRule.

Methods:

Attributes:

debugBeginRegExp property
debugBeginRegExp: str

Port of debugBeginRegExp (rule.ts:325-327).

debugName property
debugName: str

Port of Rule.debugName (rule.ts:65-68).

debugWhileRegExp property
debugWhileRegExp: str

Port of debugWhileRegExp (rule.ts:329-331).

compileWhile
compileWhile(
    grammar: IRuleFactoryHelper, endRegexSource: str | None
) -> CompiledRule

Port of compileWhile (rule.ts:361-363).

compileWhileAG
compileWhileAG(
    grammar: IRuleFactoryHelper,
    endRegexSource: str | None,
    allowA: bool,
    allowG: bool,
) -> CompiledRule

Port of compileWhileAG (rule.ts:365-367).

getContentName
getContentName(
    lineText: str,
    captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None

Port of Rule.getContentName (rule.ts:77-82).

getName
getName(
    lineText: str | None,
    captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None

Port of Rule.getName (rule.ts:70-75): expands capture references in the name against the matched text; non-capturing names pass through.

getWhileWithResolvedBackReferences
getWhileWithResolvedBackReferences(
    lineText: str,
    captureIndices: Sequence[IOnigCaptureIndex],
) -> str

Port of getWhileWithResolvedBackReferences (rule.ts:333-335): the while pattern with backreferences substituted from the begin match.

CaptureRule

CaptureRule(
    location: ILocation,
    id: int,
    name: str | None,
    contentName: str | None,
    retokenizeCapturedWithRuleId: int,
)

              flowchart TD
              textmate_grammar.engine.rules.CaptureRule[CaptureRule]
              textmate_grammar.engine.rules.Rule[Rule]

                              textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.CaptureRule
                


              click textmate_grammar.engine.rules.CaptureRule href "" "textmate_grammar.engine.rules.CaptureRule"
              click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
            

Port of CaptureRule (rule.ts:96-120) — one entry of a captures map.

retokenizeCapturedWithRuleId is 0 (no nested rule) unless the capture declares patterns, in which case it is the id of the rule compiled from them (rule.ts:476-480).

Methods:

  • dispose –

    Port of Rule.dispose (rule.ts:63); the base rule holds nothing.

  • getContentName –

    Port of Rule.getContentName (rule.ts:77-82).

  • getName –

    Port of Rule.getName (rule.ts:70-75): expands capture references in

Attributes:

  • debugName (str) –

    Port of Rule.debugName (rule.ts:65-68).

debugName property
debugName: str

Port of Rule.debugName (rule.ts:65-68).

dispose
dispose() -> None

Port of Rule.dispose (rule.ts:63); the base rule holds nothing.

Subclasses override to drop their compiled-pattern caches (kept for parity; garbage collection makes it a cache-buster only).

getContentName
getContentName(
    lineText: str,
    captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None

Port of Rule.getContentName (rule.ts:77-82).

getName
getName(
    lineText: str | None,
    captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None

Port of Rule.getName (rule.ts:70-75): expands capture references in the name against the matched text; non-capturing names pass through.

CompilePatternsResult dataclass

CompilePatternsResult(
    patterns: tuple[int, ...], hasMissingPatterns: bool
)

Port of ICompilePatternsResult (rule.ts:91-94): the ordered rule ids of a pattern list plus upstream's flag recording that entries were dropped (include targets that could not be resolved).

IGrammarRegistry

Port of IGrammarRegistry (rule.ts:36-38): resolves scope-name includes against grammars looked up by scope name. The concrete implementation arrives with the registry layer (task 4b); here it is only the hook shape.

Methods:

get_external_grammar
get_external_grammar(
    scope_name: str, repository: IRawRepositoryDict | None
) -> IRawGrammarDict | None

Port of getExternalGrammar(scopeName, repository) (rule.ts:37).

IRuleFactoryHelper


              flowchart TD
              textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
              textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
              textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]

                              textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                
                textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                


              click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
              click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
              click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
            

Port of IRuleFactoryHelper (rule.ts:40-41). Upstream Grammar additionally implements IOnigLib (grammar.ts:98), so the helper doubles as the onig-lib argument of the compile calls; the two createOnig* members come from Grammar.createOnigScanner/createOnigString (grammar.ts:177-181).

Methods:

create_onig_scanner
create_onig_scanner(sources: Sequence[str]) -> Any

Port of createOnigScanner (grammar.ts:177-179).

create_onig_string
create_onig_string(string: str) -> Any

Port of createOnigString (grammar.ts:180-181).

get_external_grammar
get_external_grammar(
    scope_name: str, repository: IRawRepositoryDict | None
) -> IRawGrammarDict | None

Port of getExternalGrammar(scopeName, repository) (rule.ts:37).

get_rule
get_rule(rule_id: int) -> Rule

Port of getRule(ruleId) (rule.ts:32).

register_rule
register_rule(factory: Callable[[int], T]) -> T

Port of registerRule(factory) (rule.ts:33): the factory is invoked immediately with the freshly allocated id.

IRuleRegistry

Port of IRuleRegistry (rule.ts:31-34): numeric rule registration and lookup by id.

Methods:

  • get_rule –

    Port of getRule(ruleId) (rule.ts:32).

  • register_rule –

    Port of registerRule(factory) (rule.ts:33): the factory is invoked

get_rule
get_rule(rule_id: int) -> Rule

Port of getRule(ruleId) (rule.ts:32).

register_rule
register_rule(factory: Callable[[int], T]) -> T

Port of registerRule(factory) (rule.ts:33): the factory is invoked immediately with the freshly allocated id.

IncludeOnlyRule

IncludeOnlyRule(
    location: ILocation,
    id: int,
    name: str | None,
    contentName: str | None,
    patterns: CompilePatternsResult,
)

              flowchart TD
              textmate_grammar.engine.rules.IncludeOnlyRule[IncludeOnlyRule]
              textmate_grammar.engine.rules.Rule[Rule]

                              textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.IncludeOnlyRule
                


              click textmate_grammar.engine.rules.IncludeOnlyRule href "" "textmate_grammar.engine.rules.IncludeOnlyRule"
              click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
            

Port of IncludeOnlyRule (rule.ts:166-207): a rule made purely of include references whose own (possibly missing) patterns are collected at compile time.

Methods:

  • getContentName –

    Port of Rule.getContentName (rule.ts:77-82).

  • getName –

    Port of Rule.getName (rule.ts:70-75): expands capture references in

Attributes:

  • debugName (str) –

    Port of Rule.debugName (rule.ts:65-68).

debugName property
debugName: str

Port of Rule.debugName (rule.ts:65-68).

getContentName
getContentName(
    lineText: str,
    captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None

Port of Rule.getContentName (rule.ts:77-82).

getName
getName(
    lineText: str | None,
    captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None

Port of Rule.getName (rule.ts:70-75): expands capture references in the name against the matched text; non-capturing names pass through.

IncludeReference dataclass

IncludeReference(
    kind: IncludeReferenceKind,
    scopeName: str | None = None,
    ruleName: str | None = None,
)

Port of the IncludeReference union (grammarDependencies.ts:240-276); fields beyond kind are set per kind (scope_name/rule_name).

IncludeReferenceKind


              flowchart TD
              textmate_grammar.engine.rules.IncludeReferenceKind[IncludeReferenceKind]

              

              click textmate_grammar.engine.rules.IncludeReferenceKind href "" "textmate_grammar.engine.rules.IncludeReferenceKind"
            

Port of IncludeReferenceKind (grammarDependencies.ts:247-253).

MatchRule

MatchRule(
    location: ILocation,
    id: int,
    name: str | None,
    match: str,
    captures: list[CaptureRule | None],
)

              flowchart TD
              textmate_grammar.engine.rules.MatchRule[MatchRule]
              textmate_grammar.engine.rules.Rule[Rule]

                              textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.MatchRule
                


              click textmate_grammar.engine.rules.MatchRule href "" "textmate_grammar.engine.rules.MatchRule"
              click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
            

Port of MatchRule (rule.ts:122-164): a single match pattern with optional captures.

Methods:

  • getContentName –

    Port of Rule.getContentName (rule.ts:77-82).

  • getName –

    Port of Rule.getName (rule.ts:70-75): expands capture references in

Attributes:

debugMatchRegExp property
debugMatchRegExp: str

Port of debugMatchRegExp (rule.ts:141-143).

debugName property
debugName: str

Port of Rule.debugName (rule.ts:65-68).

getContentName
getContentName(
    lineText: str,
    captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None

Port of Rule.getContentName (rule.ts:77-82).

getName
getName(
    lineText: str | None,
    captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None

Port of Rule.getName (rule.ts:70-75): expands capture references in the name against the matched text; non-capturing names pass through.

Rule

Rule(
    location: ILocation,
    id: int,
    name: str | None,
    contentName: str | None,
)

Port of the abstract Rule (rule.ts:43-89): identity, name/ contentName with capture expansion, and the pattern collection / compile contract the tokenization loop drives.

Methods:

  • collectPatterns –

    Port of Rule.collectPatterns (rule.ts:84) — abstract.

  • compile –

    Port of Rule.compile (rule.ts:86) — abstract.

  • compileAG –

    Port of Rule.compileAG (rule.ts:88) — abstract.

  • dispose –

    Port of Rule.dispose (rule.ts:63); the base rule holds nothing.

  • getContentName –

    Port of Rule.getContentName (rule.ts:77-82).

  • getName –

    Port of Rule.getName (rule.ts:70-75): expands capture references in

Attributes:

  • debugName (str) –

    Port of Rule.debugName (rule.ts:65-68).

debugName property
debugName: str

Port of Rule.debugName (rule.ts:65-68).

collectPatterns
collectPatterns(
    grammar: IRuleRegistry, out: RegExpSourceList
) -> None

Port of Rule.collectPatterns (rule.ts:84) — abstract.

compile
compile(
    grammar: IRuleFactoryHelper, endRegexSource: str | None
) -> CompiledRule

Port of Rule.compile (rule.ts:86) — abstract.

compileAG
compileAG(
    grammar: IRuleFactoryHelper,
    endRegexSource: str | None,
    allowA: bool,
    allowG: bool,
) -> CompiledRule

Port of Rule.compileAG (rule.ts:88) — abstract.

dispose
dispose() -> None

Port of Rule.dispose (rule.ts:63); the base rule holds nothing.

Subclasses override to drop their compiled-pattern caches (kept for parity; garbage collection makes it a cache-buster only).

getContentName
getContentName(
    lineText: str,
    captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None

Port of Rule.getContentName (rule.ts:77-82).

getName
getName(
    lineText: str | None,
    captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None

Port of Rule.getName (rule.ts:70-75): expands capture references in the name against the matched text; non-capturing names pass through.

RuleFactory

Port of RuleFactory (rule.ts:381-573).

Methods:

compile_captures staticmethod
compile_captures(
    captures: IRawRuleDict | None,
    helper: IRuleFactoryHelper,
    repository: IRawRepositoryDict,
) -> list[CaptureRule | None]

Port of _compileCaptures (rule.ts:449-485): builds the (CaptureRule | None)[] indexed by capture group number 1..n.

compile_patterns staticmethod
compile_patterns(
    patterns: list[IRawRuleDict] | None,
    helper: IRuleFactoryHelper,
    repository: IRawRepositoryDict,
) -> CompilePatternsResult

Port of _compilePatterns (rule.ts:487-572): compiles the raw pattern list to rule ids, preserving order, resolving include entries and dropping rules whose include chain resolves to nothing instantiated.

create_capture_rule staticmethod
create_capture_rule(
    helper: IRuleFactoryHelper,
    location: ILocation,
    name: str | None,
    contentName: str | None,
    retokenizeCapturedWithRuleId: int,
) -> CaptureRule

Port of createCaptureRule (rule.ts:383-387): registers the capture rule under a fresh id.

get_compiled_rule_id staticmethod
get_compiled_rule_id(
    desc: IRawRuleDict,
    helper: IRuleFactoryHelper,
    repository: IRawRepositoryDict,
) -> int

Port of getCompiledRuleId (rule.ts:389-447): the dispatch on the raw rule's shape, memoized on the rule dict via its internal id field.

Note on recursion: upstream writes desc.id = id inside the factory callback, which is invoked synchronously by registerRule — so the id is stored before any nested getCompiledRuleId runs, exactly as here.

RuleRegistry

RuleRegistry(onig_lib: Any = None)

              flowchart TD
              textmate_grammar.engine.rules.RuleRegistry[RuleRegistry]
              textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
              textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
              textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]

                              textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.rules.RuleRegistry
                                textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                
                textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                



              click textmate_grammar.engine.rules.RuleRegistry href "" "textmate_grammar.engine.rules.RuleRegistry"
              click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
              click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
              click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
            

Port of the rule-id <-> Rule bookkeeping that lives in Grammar (grammar.ts:100, 127-128, 245-254): monotonically increasing numeric ids starting at 1, with the rule objects stored under their id. Serves as the concrete :class:IRuleFactoryHelper for :class:RuleFactory users and, later (task 3), as the registry component of Grammar.

Note: upstream v9.3.2 has no standalone RuleRegistry class and no scope-name interning; the class here is simply the extracted Grammar behaviour.

Methods:

  • create_onig_scanner –

    Port of Grammar.createOnigScanner (grammar.ts:177-179).

  • create_onig_string –

    Port of Grammar.createOnigString (grammar.ts:180-181).

  • get_external_grammar –

    Port of Grammar.getExternalGrammar (grammar.ts:256-274): scope-name

  • get_rule –

    Port of Grammar.getRule (grammar.ts:252-254). May return the

  • register_rule –

    Port of Grammar.registerRule (grammar.ts:245-250).

create_onig_scanner
create_onig_scanner(sources: Sequence[str]) -> Any

Port of Grammar.createOnigScanner (grammar.ts:177-179).

create_onig_string
create_onig_string(string: str) -> Any

Port of Grammar.createOnigString (grammar.ts:180-181).

get_external_grammar
get_external_grammar(
    scope_name: str, repository: IRawRepositoryDict | None
) -> IRawGrammarDict | None

Port of Grammar.getExternalGrammar (grammar.ts:256-274): scope-name includes resolve through the grammar repository; without a registry layer behind this helper there is nothing to resolve (task 4b provides it).

get_rule
get_rule(rule_id: int) -> Rule

Port of Grammar.getRule (grammar.ts:252-254). May return the (python) None of a not-yet-registered rule while its factory is still running — upstream yields undefined there, and the only reader that can act on it treats non-None like undefined (see RuleFactory.compile_patterns).

register_rule
register_rule(factory: Callable[[int], T]) -> T

Port of Grammar.registerRule (grammar.ts:245-250).

create_rule

create_rule(
    desc: IRawRuleDict,
    helper: IRuleFactoryHelper,
    repository: IRawRepositoryDict,
) -> int

Mirror of RuleFactory.getCompiledRuleId (task-card spelling); prefer :meth:RuleFactory.get_compiled_rule_id.

init_grammar

init_grammar(
    grammar: IRawGrammarDict,
    base: IRawRuleDict | None = None,
) -> IRawGrammarDict

Port of initGrammar (grammar.ts:410-421): deep-copies the raw grammar and installs the $self and $base repository entries the include resolution relies on ($self = the top-level patterns, $base = the caller-provided base grammar's self entry or the grammar's own).

parse_include

parse_include(include: str) -> IncludeReference

Port of parseInclude (grammarDependencies.ts:278-295): $base / $self, #rule (relative), scope (top level), scope#rule (top-level repository reference). Note: this is the shared include parser; the full grammarDependency machinery (task 4b) should import it from here.

tokenize

The vscode-textmate v9.3.2 state model and line tokenization engine.

The main loop follows grammar/tokenizeString.ts. Scope names and state are kept as immutable parent-linked structures, while enter/anchor positions are line-local mutable bookkeeping exactly as in StateStackImpl.

Classes:

  • AttributedScopeStack –

    Scope path node plus vscode-textmate's inherited token/style attributes.

  • BalancedBracketSelectors –

    Balanced-bracket selector matcher for the optional binary-token API.

  • Grammar –

    A loaded grammar and its vscode-textmate line tokenizer.

  • LineTokens –

    Minimal LineTokens implementation for scope-array tokens.

  • ScopeStack –

    The theme.ts linked scope path, in root-first observable order.

  • StateStackImpl –

    Port of grammar.ts StateStackImpl (including line-scoped positions).

Functions:

  • create_grammar –

    Construct a grammar; external_grammar is the registry integration seam.

AttributedScopeStack dataclass

AttributedScopeStack(
    parent: AttributedScopeStack | None,
    scope_path: ScopeStack,
    token_attributes: int = 0,
    font_attributes: Any = None,
    style_attributes: Any = None,
)

Scope path node plus vscode-textmate's inherited token/style attributes.

BalancedBracketSelectors

BalancedBracketSelectors(
    balanced: Sequence[str], unbalanced: Sequence[str]
)

Balanced-bracket selector matcher for the optional binary-token API.

Grammar

Grammar(
    scope_name: str,
    grammar: Mapping[str, Any],
    initial_language: int = 1,
    embedded_languages: Mapping[str, int] | None = None,
    token_types: Mapping[str, int] | None = None,
    balanced_bracket_selectors: BalancedBracketSelectors
    | None = None,
    grammar_repository: Any = None,
    onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
    *,
    external_grammar: Callable[
        [str], Mapping[str, Any] | None
    ]
    | None = None,
)

              flowchart TD
              textmate_grammar.engine.tokenize.Grammar[Grammar]
              textmate_grammar.engine.rules.RuleRegistry[RuleRegistry]
              textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
              textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
              textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]

                              textmate_grammar.engine.rules.RuleRegistry --> textmate_grammar.engine.tokenize.Grammar
                                textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.rules.RuleRegistry
                                textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                
                textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                


                textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.tokenize.Grammar
                                textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                
                textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
                



              click textmate_grammar.engine.tokenize.Grammar href "" "textmate_grammar.engine.tokenize.Grammar"
              click textmate_grammar.engine.rules.RuleRegistry href "" "textmate_grammar.engine.rules.RuleRegistry"
              click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
              click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
              click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
            

A loaded grammar and its vscode-textmate line tokenizer.

Methods:

  • create_onig_scanner –

    Port of Grammar.createOnigScanner (grammar.ts:177-179).

  • create_onig_string –

    Port of Grammar.createOnigString (grammar.ts:180-181).

  • get_rule –

    Port of Grammar.getRule (grammar.ts:252-254). May return the

  • register_rule –

    Port of Grammar.registerRule (grammar.ts:245-250).

create_onig_scanner
create_onig_scanner(sources: Sequence[str]) -> Any

Port of Grammar.createOnigScanner (grammar.ts:177-179).

create_onig_string
create_onig_string(string: str) -> Any

Port of Grammar.createOnigString (grammar.ts:180-181).

get_rule
get_rule(rule_id: int) -> Rule

Port of Grammar.getRule (grammar.ts:252-254). May return the (python) None of a not-yet-registered rule while its factory is still running — upstream yields undefined there, and the only reader that can act on it treats non-None like undefined (see RuleFactory.compile_patterns).

register_rule
register_rule(factory: Callable[[int], T]) -> T

Port of Grammar.registerRule (grammar.ts:245-250).

LineTokens

LineTokens(line_text: str)

Minimal LineTokens implementation for scope-array tokens.

ScopeStack

ScopeStack(parent: ScopeStack | None, scope_name: str)

The theme.ts linked scope path, in root-first observable order.

StateStackImpl

StateStackImpl(
    parent: StateStackImpl | None,
    rule_id: int,
    enter_pos: int,
    anchor_pos: int,
    begin_rule_captured_eol: bool,
    end_rule: str | None,
    name_scopes_list: AttributedScopeStack | None,
    content_name_scopes_list: AttributedScopeStack | None,
)

Port of grammar.ts StateStackImpl (including line-scoped positions).

create_grammar

create_grammar(
    scope_name: str,
    grammar: Mapping[str, Any],
    initial_language: int = 1,
    embedded_languages: Mapping[str, int] | None = None,
    token_types: Mapping[str, int] | None = None,
    balanced_bracket_selectors: BalancedBracketSelectors
    | None = None,
    grammar_repository: Any = None,
    onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
    *,
    external_grammar: Callable[
        [str], Mapping[str, Any] | None
    ]
    | None = None,
) -> Grammar

Construct a grammar; external_grammar is the registry integration seam.

parsers

Modules:

  • markdown –

    Small Markdown convenience wrapper around the public grammar engine.

  • matlab –

    Small MATLAB convenience wrapper around the public grammar engine.

markdown

Small Markdown convenience wrapper around the public grammar engine.

Functions:

markdown_grammar cached

markdown_grammar() -> Grammar

Return the module-cached Markdown grammar.

tokenize_markdown

tokenize_markdown(
    path_or_text: str | Path,
) -> list[list[IToken]]

Tokenize a Markdown source path or text, returning tokens grouped by line.

matlab

Small MATLAB convenience wrapper around the public grammar engine.

Functions:

  • matlab_grammar –

    Return the module-cached MATLAB grammar.

  • tokenize_matlab –

    Tokenize a MATLAB source path or text, returning tokens grouped by line.

matlab_grammar cached

matlab_grammar() -> Grammar

Return the module-cached MATLAB grammar.

tokenize_matlab

tokenize_matlab(
    path_or_text: str | Path,
) -> list[list[IToken]]

Tokenize a MATLAB source path or text, returning tokens grouped by line.