API reference¶
The public API mirrors vscode-textmate's registry and stateful tokenization model.
textmate_grammar
¶
Python port of vscode-textmate's grammar tokenizer.
Modules:
Classes:
-
Grammar–A loaded grammar and its vscode-textmate line tokenizer.
-
Registry–Registry options, dependency resolution, and lazy grammar construction.
-
RegistryOptions–Synchronous counterpart of upstream
RegistryOptions. -
SyncRegistry–Synchronous equivalent of upstream
SyncRegistry.
Functions:
-
create_grammar–Construct a grammar;
external_grammaris the registry integration seam. -
parse_raw_grammar–Parse a grammar from
content(parseRawGrammar.tsupstream).
Grammar
¶
Grammar(
scope_name: str,
grammar: Mapping[str, Any],
initial_language: int = 1,
embedded_languages: Mapping[str, int] | None = None,
token_types: Mapping[str, int] | None = None,
balanced_bracket_selectors: BalancedBracketSelectors
| None = None,
grammar_repository: Any = None,
onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
*,
external_grammar: Callable[
[str], Mapping[str, Any] | None
]
| None = None,
)
flowchart TD
textmate_grammar.Grammar[Grammar]
textmate_grammar.engine.rules.RuleRegistry[RuleRegistry]
textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]
textmate_grammar.engine.rules.RuleRegistry --> textmate_grammar.Grammar
textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.rules.RuleRegistry
textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.Grammar
textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
click textmate_grammar.Grammar href "" "textmate_grammar.Grammar"
click textmate_grammar.engine.rules.RuleRegistry href "" "textmate_grammar.engine.rules.RuleRegistry"
click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
A loaded grammar and its vscode-textmate line tokenizer.
Methods:
-
create_onig_scanner–Port of
Grammar.createOnigScanner(grammar.ts:177-179). -
create_onig_string–Port of
Grammar.createOnigString(grammar.ts:180-181). -
get_rule–Port of
Grammar.getRule(grammar.ts:252-254). May return the -
register_rule–Port of
Grammar.registerRule(grammar.ts:245-250).
create_onig_scanner
¶
Port of Grammar.createOnigScanner (grammar.ts:177-179).
create_onig_string
¶
Port of Grammar.createOnigString (grammar.ts:180-181).
get_rule
¶
Port of Grammar.getRule (grammar.ts:252-254). May return the
(python) None of a not-yet-registered rule while its factory is still
running — upstream yields undefined there, and the only reader that
can act on it treats non-None like undefined (see
RuleFactory.compile_patterns).
Registry
¶
Registry(
options: RegistryOptions,
*,
grammar_factory: GrammarFactory | None = None,
)
Registry options, dependency resolution, and lazy grammar construction.
Methods:
-
addGrammar–Register and construct a grammar, matching upstream's convenience method.
-
loadGrammar–vscode-textmate spelling of :meth:
load_grammar. -
load_grammar–Load a grammar and every referenced grammar before constructing it.
-
register_grammar–Register an already parsed raw grammar and optional injection scopes.
addGrammar
¶
addGrammar(
raw_grammar: IRawGrammar,
injections: Sequence[str] = (),
initial_language: int = 0,
embedded_languages: Mapping[str, int] | None = None,
) -> TokenizerLike
Register and construct a grammar, matching upstream's convenience method.
loadGrammar
¶
loadGrammar(
initial_scope_name: str,
) -> TokenizerLike | None
vscode-textmate spelling of :meth:load_grammar.
load_grammar
¶
load_grammar(
initial_scope_name: str,
) -> TokenizerLike | None
Load a grammar and every referenced grammar before constructing it.
RegistryOptions
¶
flowchart TD
textmate_grammar.RegistryOptions[RegistryOptions]
click textmate_grammar.RegistryOptions href "" "textmate_grammar.RegistryOptions"
Synchronous counterpart of upstream RegistryOptions.
SyncRegistry
¶
SyncRegistry(onig_lib: IOnigLib = DEFAULT_ONIG_LIB)
Synchronous equivalent of upstream SyncRegistry.
Methods:
-
get_external_grammar–Look up and initialize an included grammar, cached by scope and base.
get_external_grammar
¶
get_external_grammar(
scope_name: str,
repository: Mapping[str, Any] | None = None,
) -> IRawGrammar | None
Look up and initialize an included grammar, cached by scope and base.
Upstream caches per concrete Grammar instance. The registry-level
substitute keys by the caller's $base object as well, preventing one
root grammar's base repository from leaking into another.
create_grammar
¶
create_grammar(
scope_name: str,
grammar: Mapping[str, Any],
initial_language: int = 1,
embedded_languages: Mapping[str, int] | None = None,
token_types: Mapping[str, int] | None = None,
balanced_bracket_selectors: BalancedBracketSelectors
| None = None,
grammar_repository: Any = None,
onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
*,
external_grammar: Callable[
[str], Mapping[str, Any] | None
]
| None = None,
) -> Grammar
Construct a grammar; external_grammar is the registry integration seam.
parse_raw_grammar
¶
Parse a grammar from content (parseRawGrammar.ts upstream).
Routing is by file path only: paths ending in .json parse as JSON,
everything else (including .jsonc, .yaml and .xml) parses as
plist — upstream supports nothing else.
engine
¶
Public vscode-textmate-shaped tokenization API.
Modules:
-
debug–Debug flags ported from vscode-textmate's
src/debug.ts. -
matcher–Port of vscode-textmate's
src/matcher.ts. -
oniglib–Port of vscode-textmate's
src/onigLib.ts— the oniguruma library adapter. -
plist–Faithful port of microsoft/vscode-textmate
src/plist.ts(v9.3.2). -
raw_grammar–Faithful port of microsoft/vscode-textmate
src/rawGrammar.ts, -
regexsource–Port of the
RegExpSourcemachinery from vscode-textmate'ssrc/rule.ts. -
registry–Grammar registration and dependency loading from vscode-textmate v9.3.2.
-
rules–Port of the
Ruleclasses,RuleRegistryand the rule factory from -
tokenize–The vscode-textmate v9.3.2 state model and line tokenization engine.
Classes:
-
FindOption–Port of the
FindOptionconst enum fromonigLib.ts(onigLib.ts:23-41). -
Grammar–A loaded grammar and its vscode-textmate line tokenizer.
-
ILocation–Source location of a grammar construct (
ILocationupstream). -
IOnigCaptureIndex–Port of
IOnigCaptureIndex(onigLib.ts:12-16). -
IOnigLib–Port of the
IOnigLibinterface (onigLib.ts:7-10): the library surface -
IOnigMatch–Port of
IOnigMatch(onigLib.ts:18-21): the index of the matching pattern -
OnigLib–The concrete onigurumacffi-backed :class:
IOnigLib— the role -
OnigScanner–Port of
OnigScannerfrom vscode-oniguruma — the scanner behind -
OnigString–Port of
OnigString(onigLib.ts:48-51). -
Registry–Registry options, dependency resolution, and lazy grammar construction.
-
RegistryOptions–Synchronous counterpart of upstream
RegistryOptions. -
StateStackImpl–Port of grammar.ts
StateStackImpl(including line-scoped positions). -
SyncRegistry–Synchronous equivalent of upstream
SyncRegistry.
Functions:
-
create_grammar–Construct a grammar;
external_grammaris the registry integration seam. -
disposeOnigString–Port of
disposeOnigString(onigLib.ts:53-57). -
parse_raw_grammar–Parse a grammar from
content(parseRawGrammar.tsupstream).
FindOption
¶
flowchart TD
textmate_grammar.engine.FindOption[FindOption]
click textmate_grammar.engine.FindOption href "" "textmate_grammar.engine.FindOption"
Port of the FindOption const enum from onigLib.ts (onigLib.ts:23-41).
The bit values are vscode-textmate's own numbering; they are translated to the
oniguruma search options named in the upstream member comments
(:func:_translate_options). DEBUG_CALL is a debugging marker only and is
dropped, like the debugCall flag is in vscode-oniguruma's non-debug path.
Grammar
¶
Grammar(
scope_name: str,
grammar: Mapping[str, Any],
initial_language: int = 1,
embedded_languages: Mapping[str, int] | None = None,
token_types: Mapping[str, int] | None = None,
balanced_bracket_selectors: BalancedBracketSelectors
| None = None,
grammar_repository: Any = None,
onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
*,
external_grammar: Callable[
[str], Mapping[str, Any] | None
]
| None = None,
)
flowchart TD
textmate_grammar.engine.Grammar[Grammar]
textmate_grammar.engine.rules.RuleRegistry[RuleRegistry]
textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]
textmate_grammar.engine.rules.RuleRegistry --> textmate_grammar.engine.Grammar
textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.rules.RuleRegistry
textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.Grammar
textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
click textmate_grammar.engine.Grammar href "" "textmate_grammar.engine.Grammar"
click textmate_grammar.engine.rules.RuleRegistry href "" "textmate_grammar.engine.rules.RuleRegistry"
click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
A loaded grammar and its vscode-textmate line tokenizer.
Methods:
-
create_onig_scanner–Port of
Grammar.createOnigScanner(grammar.ts:177-179). -
create_onig_string–Port of
Grammar.createOnigString(grammar.ts:180-181). -
get_rule–Port of
Grammar.getRule(grammar.ts:252-254). May return the -
register_rule–Port of
Grammar.registerRule(grammar.ts:245-250).
create_onig_scanner
¶
Port of Grammar.createOnigScanner (grammar.ts:177-179).
create_onig_string
¶
Port of Grammar.createOnigString (grammar.ts:180-181).
get_rule
¶
Port of Grammar.getRule (grammar.ts:252-254). May return the
(python) None of a not-yet-registered rule while its factory is still
running — upstream yields undefined there, and the only reader that
can act on it treats non-None like undefined (see
RuleFactory.compile_patterns).
ILocation
¶
flowchart TD
textmate_grammar.engine.ILocation[ILocation]
click textmate_grammar.engine.ILocation href "" "textmate_grammar.engine.ILocation"
Source location of a grammar construct (ILocation upstream).
IOnigCaptureIndex
dataclass
¶
Port of IOnigCaptureIndex (onigLib.ts:12-16).
IOnigLib
¶
flowchart TD
textmate_grammar.engine.IOnigLib[IOnigLib]
click textmate_grammar.engine.IOnigLib href "" "textmate_grammar.engine.IOnigLib"
Port of the IOnigLib interface (onigLib.ts:7-10): the library surface
vscode-textmate expects the tokenization engine to be wired against.
Methods:
-
create_onig_scanner–Port of
createOnigScanner(sources). -
create_onig_string–Port of
createOnigString(str).
create_onig_scanner
¶
create_onig_scanner(sources: Sequence[str]) -> OnigScanner
Port of createOnigScanner(sources).
IOnigMatch
dataclass
¶
IOnigMatch(
index: int,
capture_indices: tuple[IOnigCaptureIndex, ...],
)
Port of IOnigMatch (onigLib.ts:18-21): the index of the matching pattern
within the scanner and the captured regions of that match.
OnigLib
¶
The concrete onigurumacffi-backed :class:IOnigLib — the role
vscode-oniguruma plays upstream when main.ts wires up the default library.
OnigScanner
¶
Port of OnigScanner from vscode-oniguruma — the scanner behind
IOnigLib.createOnigScanner (onigLib.ts:43-46).
Patterns are compiled eagerly at construction, like vscode-oniguruma does; an
invalid pattern raises :class:onigurumacffi.OnigError where upstream throws
the oniguruma error message. Matching uses one RegSet search per call, whose
ONIG_REGSET_POSITION_LEAD strategy yields the same candidate selection as
vscode-oniguruma's C++ findNextOnigScannerMatch: the match beginning at the
smallest offset wins and ties break toward the lowest pattern index.
Methods:
-
dispose–Port of the optional
disposemember: drops the compiled pattern set. -
find_best_match–Convenience mirroring the pre-4.0 ``IOnigScanner.findBestMatch(position,
-
find_next_match_sync–Port of
OnigScanner.findNextMatchSync(onigLib.ts:44).
find_best_match
¶
find_best_match(
position: int,
line: str,
has_anchor: bool = False,
has_back_references: bool = False,
) -> IOnigMatch | None
Convenience mirroring the pre-4.0 IOnigScanner.findBestMatch(position,
line, hasAnchor, hasBackReferences) surface named in the porting plan.
position is the scan offset, so a \G binds there: with
has_anchor=True the patterns are scanned as-is, with has_anchor=False
FindOption.NOT_BEGIN_POSITION makes every \G fail (the same
equivalent option vscode-textmate's getFindOptions builds for
allowG=False in grammar/tokenizeString.ts:549-557).
has_back_references is bookkeeping parity for the old signature only:
in the current vscode-textmate, \n backreference patterns are resolved
before scanning (rule.ts resolveBackReferences, ported in
:mod:textmate_grammar.engine.regexsource), so the scanner has nothing to
do with them — the flag is accepted and unused.
find_next_match_sync
¶
find_next_match_sync(
string: str | OnigString,
start_position: int = 0,
options: FindOption = NONE,
) -> IOnigMatch | None
Port of OnigScanner.findNextMatchSync (onigLib.ts:44).
string may be a plain string or an :class:OnigString, as upstream.
start_position is the offset the scan starts at, which is where
\G binds: a pattern containing \G is compiled once and still
matches at whatever initial offset the caller passes.
OnigString
¶
OnigString(content: str)
Port of OnigString (onigLib.ts:48-51).
Upstream this wraps the wasm-resident UTF-8 encoding of the string and carries
the UTF-8/UTF-16 offset translation tables; Python strings are indexed by code
point natively, so this is the light wrapper keeping content around, with a
no-op dispose (parity of the optional dispose? member).
Methods:
-
dispose–No-op — there is no wasm string buffer to release.
Registry
¶
Registry(
options: RegistryOptions,
*,
grammar_factory: GrammarFactory | None = None,
)
Registry options, dependency resolution, and lazy grammar construction.
Methods:
-
addGrammar–Register and construct a grammar, matching upstream's convenience method.
-
loadGrammar–vscode-textmate spelling of :meth:
load_grammar. -
load_grammar–Load a grammar and every referenced grammar before constructing it.
-
register_grammar–Register an already parsed raw grammar and optional injection scopes.
addGrammar
¶
addGrammar(
raw_grammar: IRawGrammar,
injections: Sequence[str] = (),
initial_language: int = 0,
embedded_languages: Mapping[str, int] | None = None,
) -> TokenizerLike
Register and construct a grammar, matching upstream's convenience method.
loadGrammar
¶
loadGrammar(
initial_scope_name: str,
) -> TokenizerLike | None
vscode-textmate spelling of :meth:load_grammar.
load_grammar
¶
load_grammar(
initial_scope_name: str,
) -> TokenizerLike | None
Load a grammar and every referenced grammar before constructing it.
RegistryOptions
¶
flowchart TD
textmate_grammar.engine.RegistryOptions[RegistryOptions]
click textmate_grammar.engine.RegistryOptions href "" "textmate_grammar.engine.RegistryOptions"
Synchronous counterpart of upstream RegistryOptions.
StateStackImpl
¶
StateStackImpl(
parent: StateStackImpl | None,
rule_id: int,
enter_pos: int,
anchor_pos: int,
begin_rule_captured_eol: bool,
end_rule: str | None,
name_scopes_list: AttributedScopeStack | None,
content_name_scopes_list: AttributedScopeStack | None,
)
Port of grammar.ts StateStackImpl (including line-scoped positions).
SyncRegistry
¶
SyncRegistry(onig_lib: IOnigLib = DEFAULT_ONIG_LIB)
Synchronous equivalent of upstream SyncRegistry.
Methods:
-
get_external_grammar–Look up and initialize an included grammar, cached by scope and base.
get_external_grammar
¶
get_external_grammar(
scope_name: str,
repository: Mapping[str, Any] | None = None,
) -> IRawGrammar | None
Look up and initialize an included grammar, cached by scope and base.
Upstream caches per concrete Grammar instance. The registry-level
substitute keys by the caller's $base object as well, preventing one
root grammar's base repository from leaking into another.
create_grammar
¶
create_grammar(
scope_name: str,
grammar: Mapping[str, Any],
initial_language: int = 1,
embedded_languages: Mapping[str, int] | None = None,
token_types: Mapping[str, int] | None = None,
balanced_bracket_selectors: BalancedBracketSelectors
| None = None,
grammar_repository: Any = None,
onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
*,
external_grammar: Callable[
[str], Mapping[str, Any] | None
]
| None = None,
) -> Grammar
Construct a grammar; external_grammar is the registry integration seam.
disposeOnigString
¶
disposeOnigString(string: OnigString) -> None
Port of disposeOnigString (onigLib.ts:53-57).
parse_raw_grammar
¶
Parse a grammar from content (parseRawGrammar.ts upstream).
Routing is by file path only: paths ending in .json parse as JSON,
everything else (including .jsonc, .yaml and .xml) parses as
plist — upstream supports nothing else.
debug
¶
Debug flags ported from vscode-textmate's src/debug.ts.
Classes:
-
DebugFlags–Flags controlling optional debug behavior.
DebugFlags
¶
Flags controlling optional debug behavior.
Methods:
-
is_in_debug_mode–Read the flag at a module reload boundary, as the parser does.
matcher
¶
Port of vscode-textmate's src/matcher.ts.
Scope selectors such as
text.html.php - (meta.embedded | meta.tag), L:text.html.php meta.tag are
parsed into small composable predicate functions evaluated against a scope
path. The semantics are intentionally 1:1 with the TypeScript source; comments
throughout this module reference the corresponding sections of matcher.ts
(and of grammar.ts/theme.ts where matcher.ts delegates the name-matching
logic).
Note on ordering: the only ordering information matcher.ts itself produces
is the L: (priority -1) / R: (priority 1) selector prefix, surfaced as
:attr:MatcherWithPriority.priority. Sorting injections by that priority
happens in grammar.ts (task 3b); specificity ordering of theme rules lives
in theme.ts and is out of this plan's scope.
Classes:
-
MatcherWithPriority–matcher.ts
interface MatcherWithPriority<T> { matcher; priority }. -
ScopePath–The parent-linked scope chain matchers are evaluated against.
Functions:
-
create_matchers–Port of matcher.ts
createMatchers. -
match_scope_identifiers–Port of grammar.ts
nameMatcher, thematches_namematcher.ts is fed. -
match_scope_names–Name matcher over a :class:
ScopePathparent chain. -
scopes_are_matching–Port of grammar.ts
scopesAreMatching(matcher.ts delegates to it).
MatcherWithPriority
¶
flowchart TD
textmate_grammar.engine.matcher.MatcherWithPriority[MatcherWithPriority]
click textmate_grammar.engine.matcher.MatcherWithPriority href "" "textmate_grammar.engine.matcher.MatcherWithPriority"
matcher.ts interface MatcherWithPriority<T> { matcher; priority }.
matcher is a predicate over whatever input the caller evaluates against
(grammar.ts uses the flat scope-name list of the current token); priority
is the parsed L:/R: prefix.
ScopePath
¶
flowchart TD
textmate_grammar.engine.matcher.ScopePath[ScopePath]
click textmate_grammar.engine.matcher.ScopePath href "" "textmate_grammar.engine.matcher.ScopePath"
The parent-linked scope chain matchers are evaluated against.
A scope path is a linked chain of scope names: the node a caller holds is
the innermost scope, .name its scope name and .parent points
towards the shallower scope; the root scope has parent = None.
Materialized root-first this equals theme.ts' ScopeStack.getSegments()
(i.e. grammar.ts' getScopeNames()). Minimal on purpose; the state
stack built in task 3a satisfies this protocol structurally.
create_matchers
¶
create_matchers(
selector: str, matches_name: MatchesName[T]
) -> list[MatcherWithPriority]
Port of matcher.ts createMatchers.
Parses selector into its comma-separated branches and returns one
MatcherWithPriority per branch. Callers treat the branch matchers as
alternatives (matcher.ts consumers use matchers.some(...)). Only
priority carries ordering information; the matchers are pure predicates.
match_scope_identifiers
¶
Port of grammar.ts nameMatcher, the matches_name matcher.ts is fed.
Every identifier must occur as a subsequence of scopes, scanning
strictly left to right: identifier order matters and each identifier
consumes its position (repeated identifiers need repeated scopes). A stack
scope matches an identifier via :func:scopes_are_matching.
match_scope_names
¶
Name matcher over a :class:ScopePath parent chain.
Equivalent to grammar.ts nameMatcher fed getScopeNames(): the chain
is collected from the innermost node through the parents and then reversed
to root-first order (theme.ts ScopeStack.getSegments()).
oniglib
¶
Port of vscode-textmate's src/onigLib.ts — the oniguruma library adapter.
vscode-textmate drives tokenization through an injectable IOnigLib that provides
scanners and strings. Upstream this is backed by vscode-oniguruma (a wasm build of
oniguruma ~6.9). Here the same surface is provided on top of :mod:onigurumacffi, an
ffi binding of oniguruma 6.9 — the same engine family, so the semantics carry over:
-
Pattern selection (leftmost match wins, ties resolved by the lowest pattern index) is done by oniguruma's RegSet API (
onig_regset_searchwithONIG_REGSET_POSITION_LEAD), which is exactly the strategy vscode-oniguruma's C++findNextOnigScannerMatchuses for short strings. -
The critical semantic (see the parity plan):
\Gmatches exactly at the start offset of the scan, i.e. at thestart_positionpassed to each :meth:OnigScanner.find_next_match_synccall, not at a stored anchor. oniguruma binds\Gto the search start, and vscode-textmate passes the token offset per call —Pattern.search(content, start=start_position)reproduces this 1:1. A pattern compiled once therefore matches at different offsets for differentstart_positionvalues.
Capture indices are returned in Python code-point offsets (consistent with the rest
of the Python port; upstream returns UTF-16 offsets, having decoded the UTF-8 wasm
buffer). Unmatched capture groups — which oniguruma reports with offsets of -1 —
are mapped to start == end == 0, the same clamp vscode-oniguruma applies when its
convertUtf8OffsetToUtf16 encounters values below zero.
Every non-obvious piece references the ported section of vscode-oniguruma / onigLib.ts.
Classes:
-
FindOption–Port of the
FindOptionconst enum fromonigLib.ts(onigLib.ts:23-41). -
IOnigCaptureIndex–Port of
IOnigCaptureIndex(onigLib.ts:12-16). -
IOnigLib–Port of the
IOnigLibinterface (onigLib.ts:7-10): the library surface -
IOnigMatch–Port of
IOnigMatch(onigLib.ts:18-21): the index of the matching pattern -
OnigLib–The concrete onigurumacffi-backed :class:
IOnigLib— the role -
OnigScanner–Port of
OnigScannerfrom vscode-oniguruma — the scanner behind -
OnigString–Port of
OnigString(onigLib.ts:48-51).
Functions:
-
disposeOnigString–Port of
disposeOnigString(onigLib.ts:53-57).
FindOption
¶
flowchart TD
textmate_grammar.engine.oniglib.FindOption[FindOption]
click textmate_grammar.engine.oniglib.FindOption href "" "textmate_grammar.engine.oniglib.FindOption"
Port of the FindOption const enum from onigLib.ts (onigLib.ts:23-41).
The bit values are vscode-textmate's own numbering; they are translated to the
oniguruma search options named in the upstream member comments
(:func:_translate_options). DEBUG_CALL is a debugging marker only and is
dropped, like the debugCall flag is in vscode-oniguruma's non-debug path.
IOnigCaptureIndex
dataclass
¶
Port of IOnigCaptureIndex (onigLib.ts:12-16).
IOnigLib
¶
flowchart TD
textmate_grammar.engine.oniglib.IOnigLib[IOnigLib]
click textmate_grammar.engine.oniglib.IOnigLib href "" "textmate_grammar.engine.oniglib.IOnigLib"
Port of the IOnigLib interface (onigLib.ts:7-10): the library surface
vscode-textmate expects the tokenization engine to be wired against.
Methods:
-
create_onig_scanner–Port of
createOnigScanner(sources). -
create_onig_string–Port of
createOnigString(str).
create_onig_scanner
¶
create_onig_scanner(sources: Sequence[str]) -> OnigScanner
Port of createOnigScanner(sources).
IOnigMatch
dataclass
¶
IOnigMatch(
index: int,
capture_indices: tuple[IOnigCaptureIndex, ...],
)
Port of IOnigMatch (onigLib.ts:18-21): the index of the matching pattern
within the scanner and the captured regions of that match.
OnigLib
¶
The concrete onigurumacffi-backed :class:IOnigLib — the role
vscode-oniguruma plays upstream when main.ts wires up the default library.
OnigScanner
¶
Port of OnigScanner from vscode-oniguruma — the scanner behind
IOnigLib.createOnigScanner (onigLib.ts:43-46).
Patterns are compiled eagerly at construction, like vscode-oniguruma does; an
invalid pattern raises :class:onigurumacffi.OnigError where upstream throws
the oniguruma error message. Matching uses one RegSet search per call, whose
ONIG_REGSET_POSITION_LEAD strategy yields the same candidate selection as
vscode-oniguruma's C++ findNextOnigScannerMatch: the match beginning at the
smallest offset wins and ties break toward the lowest pattern index.
Methods:
-
dispose–Port of the optional
disposemember: drops the compiled pattern set. -
find_best_match–Convenience mirroring the pre-4.0 ``IOnigScanner.findBestMatch(position,
-
find_next_match_sync–Port of
OnigScanner.findNextMatchSync(onigLib.ts:44).
find_best_match
¶
find_best_match(
position: int,
line: str,
has_anchor: bool = False,
has_back_references: bool = False,
) -> IOnigMatch | None
Convenience mirroring the pre-4.0 IOnigScanner.findBestMatch(position,
line, hasAnchor, hasBackReferences) surface named in the porting plan.
position is the scan offset, so a \G binds there: with
has_anchor=True the patterns are scanned as-is, with has_anchor=False
FindOption.NOT_BEGIN_POSITION makes every \G fail (the same
equivalent option vscode-textmate's getFindOptions builds for
allowG=False in grammar/tokenizeString.ts:549-557).
has_back_references is bookkeeping parity for the old signature only:
in the current vscode-textmate, \n backreference patterns are resolved
before scanning (rule.ts resolveBackReferences, ported in
:mod:textmate_grammar.engine.regexsource), so the scanner has nothing to
do with them — the flag is accepted and unused.
find_next_match_sync
¶
find_next_match_sync(
string: str | OnigString,
start_position: int = 0,
options: FindOption = NONE,
) -> IOnigMatch | None
Port of OnigScanner.findNextMatchSync (onigLib.ts:44).
string may be a plain string or an :class:OnigString, as upstream.
start_position is the offset the scan starts at, which is where
\G binds: a pattern containing \G is compiled once and still
matches at whatever initial offset the caller passes.
OnigString
¶
OnigString(content: str)
Port of OnigString (onigLib.ts:48-51).
Upstream this wraps the wasm-resident UTF-8 encoding of the string and carries
the UTF-8/UTF-16 offset translation tables; Python strings are indexed by code
point natively, so this is the light wrapper keeping content around, with a
no-op dispose (parity of the optional dispose? member).
Methods:
-
dispose–No-op — there is no wasm string buffer to release.
disposeOnigString
¶
disposeOnigString(string: OnigString) -> None
Port of disposeOnigString (onigLib.ts:53-57).
plist
¶
Faithful port of microsoft/vscode-textmate src/plist.ts (v9.3.2).
A very fast dependency-free property-list (XML) parser for TextMate grammars.
It is deliberately not a general XML parser: it understands exactly the tags
TextMate grammars use (dict, array, key, string, real,
integer, date, data, true, false, plist) and skips
processing instructions, comments and doctypes. Semantics shipped here match
the TypeScript implementation 1:1, including its errors and quirks:
- Quirks preserved: closing tags and the
plisttag are matched by string prefix (</dictionary>closes as</dict>); unterminated values at EOF are accepted; there is no CDATA support (<![CDATA[...]]>is skipped to the next>like a comment); named entities only cover&<>"', while numeric entities are decimal&#NNN;and lowercase-hex&#xNNN;(uppercase hex digits are not decoded;Jstays literal);<true>/<false>ignore their text content;<integer>values always yield a JSnumber(a Pythonfloat). - Locations (
$vscodeTextmateLocationentries) are only produced in theparse_with_locationdebug path, exactly like upstream. -
Errors are
ValueErrorwith the upstream message format::Near offset
: ~~~<50 chars of input>~~~
See plist.ts in microsoft/vscode-textmate for the reference implementation.
Functions:
-
parse_plist–A very fast plist parser (
plist.parsePLISTupstream). -
parse_with_location–Parse
content, optionally embedding locations (debug mode).
raw_grammar
¶
Faithful port of microsoft/vscode-textmate src/rawGrammar.ts,
src/parseRawGrammar.ts and src/json.ts (v9.3.2).
rawGrammar.ts in the pinned upstream release is purely type-level
(IRawGrammar, IRawRule, IRawCaptures, IRawRepository,
ILocation): there is no runtime RawGrammar/RawCaptures class and no
valueAsString coercion in the pinned source (those notions date back to
pre-1.0 vscode-textmate; the modern engine simply runs on the any objects
produced by JSON.parse/plist.parsePLIST, so upstream types are runtime
lies). The upstream interfaces are mirrored here as TypedDict shapes; the
scalar coercion helper :func:value_as_string covers the one coercion
downstream rule-building code must not guess at (JS String() semantics for
string-typed fields that may technically hold plist scalars).
parseRawGrammar.ts semantics, exactly as upstream:
- a file path ending with
.json(case sensitive suffix test:/\.json$/) → JSON grammar; everything else → plist grammar. There is no content sniffing and no YAML support upstream: unknown extensions are handed to the plist parser, which fails withNear offset ...: expected <for non-XML content (preserved here). - the production path is strict JSON: upstream calls
JSON.parseand only uses the hand-rolledparseJSONtokenizer fromjson.tsin debug mode.json.loadsmirrorsJSON.parse(strict JSON: no comments, no trailing commas).
json.ts is ported in full and exported as :func:parse_json for the debug
path (VSCODE_TEXTMATE_DEBUG environment variable, standing in for
upstream DebugFlags.InDebugMode). Its tolerances — verified against the TS
source — are not jsonc: it supports neither comments nor trailing commas; its
leniency lies in JS parseFloat-style number tokens (1-2 parses as
1) and in accepting trailing garbage after the root construct.
Classes:
-
ILocation–Source location of a grammar construct (
ILocationupstream). -
JSONStreamState–Tokenizer state (
JSONStreamStateupstream).
Functions:
-
parse_json–parseJSONupstream — the hand-rolled, JS-quirky JSON tokenizer. -
parse_raw_grammar–Parse a grammar from
content(parseRawGrammar.tsupstream). -
value_as_string–Coerce a raw grammar scalar to a string with JS
String()semantics.
ILocation
¶
flowchart TD
textmate_grammar.engine.raw_grammar.ILocation[ILocation]
click textmate_grammar.engine.raw_grammar.ILocation href "" "textmate_grammar.engine.raw_grammar.ILocation"
Source location of a grammar construct (ILocation upstream).
next_json_token
¶
next_json_token(
stream_state: JSONStreamState, out_token: _JSONToken
) -> bool
Produce the next token (nextJSONToken upstream).
Precondition upstream: the source is known to be valid JSON.
Note the quirks: on False (EOS or an invalid keyword tail) the stream
state is not updated; a failing string/keyword read leaves out_token
with the type set but no value/updated state.
parse_json
¶
parseJSON upstream — the hand-rolled, JS-quirky JSON tokenizer.
Only used on the debug path upstream (VSCODE_TEXTMATE_DEBUG); embeds
$vscodeTextmateLocation when with_metadata is set.
Like the TS engine, positions are indexed by UTF-16 code units; python
values are returned with surrogate pairs recombined into astral characters
(see plist._units_of/_text_of).
parse_raw_grammar
¶
Parse a grammar from content (parseRawGrammar.ts upstream).
Routing is by file path only: paths ending in .json parse as JSON,
everything else (including .jsonc, .yaml and .xml) parses as
plist — upstream supports nothing else.
value_as_string
¶
Coerce a raw grammar scalar to a string with JS String() semantics.
Practical fallback for rule-building code that must not guess how grammar
scalars (which may technically be <integer>/<real>/<true>
values in wayward tmLanguage files) render as strings:
true/false→"true"/"false"(JS, not Python capitalization)- numbers → shortest double representation (
1.0→"1"),NaN/Infinitymatch JS tokens - everything else →
str(value)
regexsource
¶
Port of the RegExpSource machinery from vscode-textmate's src/rule.ts.
This module ports from rule.ts (version 9.3.2, the pinned reference clone):
IRegExpSourceAnchorCache (rule.ts:575-580), RegExpSource (rule.ts:582-735),
RegExpSourceList (rule.ts:744-856) and CompiledRule (rule.ts:858-894), plus
the two regex-source helpers they depend on from src/utils.ts:
escapeRegExpCharacters (utils.ts:159-161) and the RegexSource class
(utils.ts:66-115, $n/${n:/command} capture expansion used for scope names).
Semantics are 1:1 with the reference; naming is pythonic and every non-obvious piece carries a comment referencing the ported lines of rule.ts / utils.ts.
Classes:
-
AnchorCache–Port of
IRegExpSourceAnchorCache(rule.ts:575-580): the four resolved -
CompiledRule–Port of rule.ts's
CompiledRule(rule.ts:858-894): a set of pattern -
IFindNextMatchResult–Port of
IFindNextMatchResult(rule.ts:896-899): the matched pattern's -
RegExpSource–Port of rule.ts's
RegExpSource(rule.ts:582-735): a pattern source from -
RegExpSourceList–Port of rule.ts's
RegExpSourceList(rule.ts:744-856): a growing list of -
RegexSource–Port of utils.ts's static
RegexSourceclass (utils.ts:66-115): expands
Functions:
-
escapeRegExpCharacters–Port of utils.ts
escapeRegExpCharacters(utils.ts:159-161) — escapes the
AnchorCache
dataclass
¶
Port of IRegExpSourceAnchorCache (rule.ts:575-580): the four resolved
variants of a source with anchors, named after which anchors remain allowed.
A1 = \A kept, G1 = \G kept; disallowed anchors were rewritten
to the U+FFFF sentinel, which can virtually never occur in scanned text.
CompiledRule
¶
Port of rule.ts's CompiledRule (rule.ts:858-894): a set of pattern
sources compiled once into an :class:OnigScanner, resolving matches back to
the rule ids the patterns belong to.
Methods:
-
dispose–Port of
CompiledRule.dispose(rule.ts:865-869). -
find_next_match_sync–Port of
CompiledRule.findNextMatchSync(rule.ts:879-893): scans and
find_next_match_sync
¶
find_next_match_sync(
string: str | OnigString,
start_position: int,
options: FindOption = NONE,
) -> IFindNextMatchResult | None
Port of CompiledRule.findNextMatchSync (rule.ts:879-893): scans and
maps the matched pattern index back to IFindNextMatchResult.
IFindNextMatchResult
dataclass
¶
IFindNextMatchResult(
rule_id: int,
capture_indices: tuple[IOnigCaptureIndex, ...],
)
Port of IFindNextMatchResult (rule.ts:896-899): the matched pattern's
rule id and its capture indices.
RegExpSource
¶
Port of rule.ts's RegExpSource (rule.ts:582-735): a pattern source from
the grammar, preprocessed on construction (\z workaround), detecting
whether it uses anchors (\A/\G) or \n backreferences and able to
resolve the anchor variants via its anchor cache.
Walks the source once: \z (oniguruma-only, kept in tmlanguage sources)
is rewritten to a JS-compatible equivalent and occurrences of \A/\G
mark the source as anchored. Note that escaped anchors inside character
classes are flagged too — upstream does the same (rule.ts:600-613).
Methods:
-
clone–Port of
RegExpSource.clone(rule.ts:641-643). -
resolve_anchors–Port of
RegExpSource.resolveAnchors(rule.ts:716-734): the source -
resolve_back_references–Port of
RegExpSource.resolveBackReferences(rule.ts:656-664): -
set_source–Port of
RegExpSource.setSource(rule.ts:645-654): rebuilds the anchor
resolve_anchors
¶
Port of RegExpSource.resolveAnchors (rule.ts:716-734): the source
variant with the disallowed anchors neutralized; non-anchored sources are
returned verbatim (rule.ts:717-719).
resolve_back_references
¶
resolve_back_references(
line_text: str,
capture_indices: Sequence[IOnigCaptureIndex],
) -> str
Port of RegExpSource.resolveBackReferences (rule.ts:656-664):
substitutes \n-style backreferences in the source with the
regexp-escaped content of capture n from the (begin) match, e.g. for
end patterns echoing the begun content. Unmatched or out-of-range captures
substitute '' — upstream capturedValues[...] || '' (rule.ts:662).
RegExpSourceList
¶
RegExpSourceList()
Port of rule.ts's RegExpSourceList (rule.ts:744-856): a growing list of
:class:RegExpSource items compiled lazily into :class:CompiledRule objects.
For lists with anchors, the compiled scanners are cached in _anchor_cache
— upstream carries the four A0_G0..A1_G1 fields (rule.ts:737-742,
751-760); they are keyed here by (allow_a, allow_g).
Methods:
-
compile–Port of
RegExpSourceList.compile(rule.ts:812-818): compiles the -
compile_ag–Port of
RegExpSourceList.compileAG(rule.ts:820-850): non-anchored -
push–Port of
RegExpSourceList.push(rule.ts:790-793). -
set_source–Port of
RegExpSourceList.setSource(rule.ts:804-810): busts the -
unshift–Port of
RegExpSourceList.unshift(rule.ts:795-798).
compile
¶
compile(onig_lib: IOnigLib) -> CompiledRule
Port of RegExpSourceList.compile (rule.ts:812-818): compiles the
plain sources once and caches them.
compile_ag
¶
compile_ag(
onig_lib: IOnigLib, allow_a: bool, allow_g: bool
) -> CompiledRule
Port of RegExpSourceList.compileAG (rule.ts:820-850): non-anchored
lists share the plain compile (rule.ts:821-822); anchored lists get one
cached compiled variant per allowed anchor combination.
set_source
¶
Port of RegExpSourceList.setSource (rule.ts:804-810): busts the
caches when the source of an item actually changes.
RegexSource
¶
Port of utils.ts's static RegexSource class (utils.ts:66-115): expands
$n and ${n:/downcase|upcase} captures in scope names against the text
a pattern captured.
Methods:
-
hasCaptures–Port of
RegexSource.hasCaptures(utils.ts:68-75). -
replaceCaptures–Port of
RegexSource.replaceCaptures(utils.ts:77-115): replaces each
hasCaptures
staticmethod
¶
Port of RegexSource.hasCaptures (utils.ts:68-75).
replaceCaptures
staticmethod
¶
replaceCaptures(
regex_source: str,
capture_source: str,
capture_indices: Sequence[IOnigCaptureIndex],
) -> str
Port of RegexSource.replaceCaptures (utils.ts:77-115): replaces each
$n/${n:/command} occurrence with the content capture n covers in
capture_source, applying the optional case command. References to
captures that don't exist are left verbatim. Unmatched captures (oniguruma
-1 offsets) contribute '' via the empty substring slice.
registry
¶
Grammar registration and dependency loading from vscode-textmate v9.3.2.
The upstream Registry and SyncRegistry create the concrete Grammar
class. That class is not available until task 3b, so the one deliberate seam here
is an injectable grammar_factory(location, raw_grammar). Its default raises
NotImplementedError; the later tokenizer task supplies the real constructor.
Upstream exposes asynchronous methods because its loader returns promises. This
port intentionally uses synchronous callables returning raw grammar mappings (or
None); dependency traversal, load caching, registration, and construction
otherwise retain the upstream behavior. onig_lib is carried in the options
for the eventual Grammar wiring, but is not consumed by this seam yet.
Classes:
-
ExternalReferenceCollector–Deduplicating collector; raw-rule visitation uses identity like JS Set.
-
Registry–Registry options, dependency resolution, and lazy grammar construction.
-
RegistryOptions–Synchronous counterpart of upstream
RegistryOptions. -
ScopeDependencyProcessor–Queue-based port of
ScopeDependencyProcessor(lines 59-103). -
SyncRegistry–Synchronous equivalent of upstream
SyncRegistry. -
TokenizerLike–Minimal interface of a constructed grammar owned by the registry.
-
TopLevelRepositoryRuleReference–Reference to one named rule in a grammar's top-level repository.
-
TopLevelRuleReference–Reference to an entire grammar, as in grammarDependencies.ts:15-22.
Functions:
-
collect_includes–Collect external references from a grammar's own rules and injections.
-
get_grammar_dependencies–Return all reachable full grammar scopes, including injection scopes.
ExternalReferenceCollector
¶
ExternalReferenceCollector()
Deduplicating collector; raw-rule visitation uses identity like JS Set.
Registry
¶
Registry(
options: RegistryOptions,
*,
grammar_factory: GrammarFactory | None = None,
)
Registry options, dependency resolution, and lazy grammar construction.
Methods:
-
addGrammar–Register and construct a grammar, matching upstream's convenience method.
-
loadGrammar–vscode-textmate spelling of :meth:
load_grammar. -
load_grammar–Load a grammar and every referenced grammar before constructing it.
-
register_grammar–Register an already parsed raw grammar and optional injection scopes.
addGrammar
¶
addGrammar(
raw_grammar: IRawGrammar,
injections: Sequence[str] = (),
initial_language: int = 0,
embedded_languages: Mapping[str, int] | None = None,
) -> TokenizerLike
Register and construct a grammar, matching upstream's convenience method.
loadGrammar
¶
loadGrammar(
initial_scope_name: str,
) -> TokenizerLike | None
vscode-textmate spelling of :meth:load_grammar.
load_grammar
¶
load_grammar(
initial_scope_name: str,
) -> TokenizerLike | None
Load a grammar and every referenced grammar before constructing it.
RegistryOptions
¶
flowchart TD
textmate_grammar.engine.registry.RegistryOptions[RegistryOptions]
click textmate_grammar.engine.registry.RegistryOptions href "" "textmate_grammar.engine.registry.RegistryOptions"
Synchronous counterpart of upstream RegistryOptions.
ScopeDependencyProcessor
¶
ScopeDependencyProcessor(
repo: SyncRegistry, initial_scope_name: str
)
Queue-based port of ScopeDependencyProcessor (lines 59-103).
SyncRegistry
¶
SyncRegistry(onig_lib: IOnigLib = DEFAULT_ONIG_LIB)
Synchronous equivalent of upstream SyncRegistry.
Methods:
-
get_external_grammar–Look up and initialize an included grammar, cached by scope and base.
get_external_grammar
¶
get_external_grammar(
scope_name: str,
repository: Mapping[str, Any] | None = None,
) -> IRawGrammar | None
Look up and initialize an included grammar, cached by scope and base.
Upstream caches per concrete Grammar instance. The registry-level
substitute keys by the caller's $base object as well, preventing one
root grammar's base repository from leaking into another.
TokenizerLike
¶
flowchart TD
textmate_grammar.engine.registry.TokenizerLike[TokenizerLike]
click textmate_grammar.engine.registry.TokenizerLike href "" "textmate_grammar.engine.registry.TokenizerLike"
Minimal interface of a constructed grammar owned by the registry.
Methods:
-
dispose–Release tokenizer resources, when present.
TopLevelRepositoryRuleReference
dataclass
¶
Reference to one named rule in a grammar's top-level repository.
TopLevelRuleReference
dataclass
¶
TopLevelRuleReference(scope_name: str)
Reference to an entire grammar, as in grammarDependencies.ts:15-22.
collect_includes
¶
collect_includes(
grammar: IRawGrammar,
) -> list[AbsoluteRuleReference]
Collect external references from a grammar's own rules and injections.
The result is an empty-base traversal useful for inspection. Full dependency
loading should use :func:get_grammar_dependencies, which resolves local
$base and scope references against the registry like upstream does.
get_grammar_dependencies
¶
get_grammar_dependencies(
registry: SyncRegistry, scope_name: str
) -> list[str]
Return all reachable full grammar scopes, including injection scopes.
rules
¶
Port of the Rule classes, RuleRegistry and the rule factory from
vscode-textmate's src/rule.ts (version 9.3.2, the pinned reference clone).
Ported from rule.ts: endRuleId/whileRuleId (rule.ts:17-20), the Rule base
class (rule.ts:43-89), CaptureRule (rule.ts:96-120), MatchRule
(rule.ts:122-164), IncludeOnlyRule (rule.ts:166-207), BeginEndRule
(rule.ts:209-288), BeginWhileRule (rule.ts:290-379) and the RuleFactory
(rule.ts:381-573) — i.e. createCaptureRule, getCompiledRuleId,
_compileCaptures and _compilePatterns. The registry lives in Grammar
upstream: registerRule/getRule are grammar.ts:245-254; the class
:class:RuleRegistry here extracts exactly that behaviour.
parse_include ports parseInclude and IncludeReferenceKind from
src/grammar/grammarDependencies.ts (grammarDependencies.ts:247-295) — the
minimal piece the rule factory needs for include resolution. init_grammar ports
initGrammar (grammar.ts:410-421), which installs the $self/$base
repository hooks.
Semantics are 1:1 with the reference; naming is pythonic where it can be (upstream member names are kept when the python port is simply "same words"), and every non-obvious piece carries a comment referencing the ported lines of rule.ts / grammar.ts / grammarDependencies.ts.
Classes:
-
BeginEndRule–Port of
BeginEndRule(rule.ts:209-288):begin/endwith captures -
BeginWhileRule–Port of
BeginWhileRule(rule.ts:290-379):beginplus awhile -
CaptureRule–Port of
CaptureRule(rule.ts:96-120) — one entry of acapturesmap. -
CompilePatternsResult–Port of
ICompilePatternsResult(rule.ts:91-94): the ordered rule ids of -
IRuleFactoryHelper–Port of
IRuleFactoryHelper(rule.ts:40-41). UpstreamGrammar -
IRuleRegistry–Port of
IRuleRegistry(rule.ts:31-34): numeric rule registration and -
IncludeOnlyRule–Port of
IncludeOnlyRule(rule.ts:166-207): a rule made purely of -
IncludeReference–Port of the
IncludeReferenceunion (grammarDependencies.ts:240-276); -
IncludeReferenceKind–Port of
IncludeReferenceKind(grammarDependencies.ts:247-253). -
MatchRule–Port of
MatchRule(rule.ts:122-164): a singlematchpattern with -
Rule–Port of the abstract
Rule(rule.ts:43-89): identity,name/ -
RuleFactory–Port of
RuleFactory(rule.ts:381-573). -
RuleRegistry–Port of the rule-id <-> Rule bookkeeping that lives in
Grammar
Functions:
-
create_rule–Mirror of
RuleFactory.getCompiledRuleId(task-card spelling); prefer -
init_grammar–Port of
initGrammar(grammar.ts:410-421): deep-copies the raw grammar -
parse_include–Port of
parseInclude(grammarDependencies.ts:278-295):$base/
BeginEndRule
¶
BeginEndRule(
location: ILocation,
id: int,
name: str | None,
contentName: str | None,
begin: str,
beginCaptures: list[CaptureRule | None],
end: str | None,
endCaptures: list[CaptureRule | None],
applyEndPatternLast: bool | None,
patterns: CompilePatternsResult,
)
flowchart TD
textmate_grammar.engine.rules.BeginEndRule[BeginEndRule]
textmate_grammar.engine.rules.Rule[Rule]
textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.BeginEndRule
click textmate_grammar.engine.rules.BeginEndRule href "" "textmate_grammar.engine.rules.BeginEndRule"
click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
Port of BeginEndRule (rule.ts:209-288): begin/end with captures
on both, contained patterns and applyEndPatternLast placement.
Methods:
-
getContentName–Port of
Rule.getContentName(rule.ts:77-82). -
getEndWithResolvedBackReferences–Port of
getEndWithResolvedBackReferences(rule.ts:248-250): the end -
getName–Port of
Rule.getName(rule.ts:70-75): expands capture references in
Attributes:
-
debugBeginRegExp(str) –Port of
debugBeginRegExp(rule.ts:240-242). -
debugEndRegExp(str) –Port of
debugEndRegExp(rule.ts:244-246). -
debugName(str) –Port of
Rule.debugName(rule.ts:65-68).
getContentName
¶
getContentName(
lineText: str,
captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None
Port of Rule.getContentName (rule.ts:77-82).
getEndWithResolvedBackReferences
¶
getEndWithResolvedBackReferences(
lineText: str,
captureIndices: Sequence[IOnigCaptureIndex],
) -> str
Port of getEndWithResolvedBackReferences (rule.ts:248-250): the end
pattern with backreferences substituted from the begin match.
getName
¶
getName(
lineText: str | None,
captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None
Port of Rule.getName (rule.ts:70-75): expands capture references in
the name against the matched text; non-capturing names pass through.
BeginWhileRule
¶
BeginWhileRule(
location: ILocation,
id: int,
name: str | None,
contentName: str | None,
begin: str,
beginCaptures: list[CaptureRule | None],
while_: str,
whileCaptures: list[CaptureRule | None],
patterns: CompilePatternsResult,
)
flowchart TD
textmate_grammar.engine.rules.BeginWhileRule[BeginWhileRule]
textmate_grammar.engine.rules.Rule[Rule]
textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.BeginWhileRule
click textmate_grammar.engine.rules.BeginWhileRule href "" "textmate_grammar.engine.rules.BeginWhileRule"
click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
Port of BeginWhileRule (rule.ts:290-379): begin plus a while
condition checked at the start of every line (rule id whileRuleId = -2),
with the contained patterns compiled as for BeginEndRule.
Methods:
-
compileWhile–Port of
compileWhile(rule.ts:361-363). -
compileWhileAG–Port of
compileWhileAG(rule.ts:365-367). -
getContentName–Port of
Rule.getContentName(rule.ts:77-82). -
getName–Port of
Rule.getName(rule.ts:70-75): expands capture references in -
getWhileWithResolvedBackReferences–Port of
getWhileWithResolvedBackReferences(rule.ts:333-335): the
Attributes:
-
debugBeginRegExp(str) –Port of
debugBeginRegExp(rule.ts:325-327). -
debugName(str) –Port of
Rule.debugName(rule.ts:65-68). -
debugWhileRegExp(str) –Port of
debugWhileRegExp(rule.ts:329-331).
compileWhile
¶
compileWhile(
grammar: IRuleFactoryHelper, endRegexSource: str | None
) -> CompiledRule
Port of compileWhile (rule.ts:361-363).
compileWhileAG
¶
compileWhileAG(
grammar: IRuleFactoryHelper,
endRegexSource: str | None,
allowA: bool,
allowG: bool,
) -> CompiledRule
Port of compileWhileAG (rule.ts:365-367).
getContentName
¶
getContentName(
lineText: str,
captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None
Port of Rule.getContentName (rule.ts:77-82).
getName
¶
getName(
lineText: str | None,
captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None
Port of Rule.getName (rule.ts:70-75): expands capture references in
the name against the matched text; non-capturing names pass through.
getWhileWithResolvedBackReferences
¶
getWhileWithResolvedBackReferences(
lineText: str,
captureIndices: Sequence[IOnigCaptureIndex],
) -> str
Port of getWhileWithResolvedBackReferences (rule.ts:333-335): the
while pattern with backreferences substituted from the begin match.
CaptureRule
¶
CaptureRule(
location: ILocation,
id: int,
name: str | None,
contentName: str | None,
retokenizeCapturedWithRuleId: int,
)
flowchart TD
textmate_grammar.engine.rules.CaptureRule[CaptureRule]
textmate_grammar.engine.rules.Rule[Rule]
textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.CaptureRule
click textmate_grammar.engine.rules.CaptureRule href "" "textmate_grammar.engine.rules.CaptureRule"
click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
Port of CaptureRule (rule.ts:96-120) — one entry of a captures map.
retokenizeCapturedWithRuleId is 0 (no nested rule) unless the capture
declares patterns, in which case it is the id of the rule compiled from
them (rule.ts:476-480).
Methods:
-
dispose–Port of
Rule.dispose(rule.ts:63); the base rule holds nothing. -
getContentName–Port of
Rule.getContentName(rule.ts:77-82). -
getName–Port of
Rule.getName(rule.ts:70-75): expands capture references in
Attributes:
dispose
¶
dispose() -> None
Port of Rule.dispose (rule.ts:63); the base rule holds nothing.
Subclasses override to drop their compiled-pattern caches (kept for parity; garbage collection makes it a cache-buster only).
getContentName
¶
getContentName(
lineText: str,
captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None
Port of Rule.getContentName (rule.ts:77-82).
getName
¶
getName(
lineText: str | None,
captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None
Port of Rule.getName (rule.ts:70-75): expands capture references in
the name against the matched text; non-capturing names pass through.
CompilePatternsResult
dataclass
¶
Port of ICompilePatternsResult (rule.ts:91-94): the ordered rule ids of
a pattern list plus upstream's flag recording that entries were dropped
(include targets that could not be resolved).
IGrammarRegistry
¶
Port of IGrammarRegistry (rule.ts:36-38): resolves scope-name includes
against grammars looked up by scope name. The concrete implementation arrives
with the registry layer (task 4b); here it is only the hook shape.
Methods:
-
get_external_grammar–Port of
getExternalGrammar(scopeName, repository)(rule.ts:37).
IRuleFactoryHelper
¶
flowchart TD
textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]
textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
Port of IRuleFactoryHelper (rule.ts:40-41). Upstream Grammar
additionally implements IOnigLib (grammar.ts:98), so the helper doubles as
the onig-lib argument of the compile calls; the two createOnig* members
come from Grammar.createOnigScanner/createOnigString
(grammar.ts:177-181).
Methods:
-
create_onig_scanner–Port of
createOnigScanner(grammar.ts:177-179). -
create_onig_string–Port of
createOnigString(grammar.ts:180-181). -
get_external_grammar–Port of
getExternalGrammar(scopeName, repository)(rule.ts:37). -
get_rule–Port of
getRule(ruleId)(rule.ts:32). -
register_rule–Port of
registerRule(factory)(rule.ts:33): the factory is invoked
create_onig_scanner
¶
Port of createOnigScanner (grammar.ts:177-179).
create_onig_string
¶
Port of createOnigString (grammar.ts:180-181).
IRuleRegistry
¶
Port of IRuleRegistry (rule.ts:31-34): numeric rule registration and
lookup by id.
Methods:
-
get_rule–Port of
getRule(ruleId)(rule.ts:32). -
register_rule–Port of
registerRule(factory)(rule.ts:33): the factory is invoked
IncludeOnlyRule
¶
IncludeOnlyRule(
location: ILocation,
id: int,
name: str | None,
contentName: str | None,
patterns: CompilePatternsResult,
)
flowchart TD
textmate_grammar.engine.rules.IncludeOnlyRule[IncludeOnlyRule]
textmate_grammar.engine.rules.Rule[Rule]
textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.IncludeOnlyRule
click textmate_grammar.engine.rules.IncludeOnlyRule href "" "textmate_grammar.engine.rules.IncludeOnlyRule"
click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
Port of IncludeOnlyRule (rule.ts:166-207): a rule made purely of
include references whose own (possibly missing) patterns are collected at
compile time.
Methods:
-
getContentName–Port of
Rule.getContentName(rule.ts:77-82). -
getName–Port of
Rule.getName(rule.ts:70-75): expands capture references in
Attributes:
getContentName
¶
getContentName(
lineText: str,
captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None
Port of Rule.getContentName (rule.ts:77-82).
getName
¶
getName(
lineText: str | None,
captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None
Port of Rule.getName (rule.ts:70-75): expands capture references in
the name against the matched text; non-capturing names pass through.
IncludeReference
dataclass
¶
IncludeReference(
kind: IncludeReferenceKind,
scopeName: str | None = None,
ruleName: str | None = None,
)
Port of the IncludeReference union (grammarDependencies.ts:240-276);
fields beyond kind are set per kind (scope_name/rule_name).
IncludeReferenceKind
¶
flowchart TD
textmate_grammar.engine.rules.IncludeReferenceKind[IncludeReferenceKind]
click textmate_grammar.engine.rules.IncludeReferenceKind href "" "textmate_grammar.engine.rules.IncludeReferenceKind"
Port of IncludeReferenceKind (grammarDependencies.ts:247-253).
MatchRule
¶
MatchRule(
location: ILocation,
id: int,
name: str | None,
match: str,
captures: list[CaptureRule | None],
)
flowchart TD
textmate_grammar.engine.rules.MatchRule[MatchRule]
textmate_grammar.engine.rules.Rule[Rule]
textmate_grammar.engine.rules.Rule --> textmate_grammar.engine.rules.MatchRule
click textmate_grammar.engine.rules.MatchRule href "" "textmate_grammar.engine.rules.MatchRule"
click textmate_grammar.engine.rules.Rule href "" "textmate_grammar.engine.rules.Rule"
Port of MatchRule (rule.ts:122-164): a single match pattern with
optional captures.
Methods:
-
getContentName–Port of
Rule.getContentName(rule.ts:77-82). -
getName–Port of
Rule.getName(rule.ts:70-75): expands capture references in
Attributes:
-
debugMatchRegExp(str) –Port of
debugMatchRegExp(rule.ts:141-143). -
debugName(str) –Port of
Rule.debugName(rule.ts:65-68).
getContentName
¶
getContentName(
lineText: str,
captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None
Port of Rule.getContentName (rule.ts:77-82).
getName
¶
getName(
lineText: str | None,
captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None
Port of Rule.getName (rule.ts:70-75): expands capture references in
the name against the matched text; non-capturing names pass through.
Rule
¶
Port of the abstract Rule (rule.ts:43-89): identity, name/
contentName with capture expansion, and the pattern collection / compile
contract the tokenization loop drives.
Methods:
-
collectPatterns–Port of
Rule.collectPatterns(rule.ts:84) — abstract. -
compile–Port of
Rule.compile(rule.ts:86) — abstract. -
compileAG–Port of
Rule.compileAG(rule.ts:88) — abstract. -
dispose–Port of
Rule.dispose(rule.ts:63); the base rule holds nothing. -
getContentName–Port of
Rule.getContentName(rule.ts:77-82). -
getName–Port of
Rule.getName(rule.ts:70-75): expands capture references in
Attributes:
collectPatterns
¶
collectPatterns(
grammar: IRuleRegistry, out: RegExpSourceList
) -> None
Port of Rule.collectPatterns (rule.ts:84) — abstract.
compile
¶
compile(
grammar: IRuleFactoryHelper, endRegexSource: str | None
) -> CompiledRule
Port of Rule.compile (rule.ts:86) — abstract.
compileAG
¶
compileAG(
grammar: IRuleFactoryHelper,
endRegexSource: str | None,
allowA: bool,
allowG: bool,
) -> CompiledRule
Port of Rule.compileAG (rule.ts:88) — abstract.
dispose
¶
dispose() -> None
Port of Rule.dispose (rule.ts:63); the base rule holds nothing.
Subclasses override to drop their compiled-pattern caches (kept for parity; garbage collection makes it a cache-buster only).
getContentName
¶
getContentName(
lineText: str,
captureIndices: Sequence[IOnigCaptureIndex],
) -> str | None
Port of Rule.getContentName (rule.ts:77-82).
getName
¶
getName(
lineText: str | None,
captureIndices: Sequence[IOnigCaptureIndex] | None,
) -> str | None
Port of Rule.getName (rule.ts:70-75): expands capture references in
the name against the matched text; non-capturing names pass through.
RuleFactory
¶
Port of RuleFactory (rule.ts:381-573).
Methods:
-
compile_captures–Port of
_compileCaptures(rule.ts:449-485): builds the -
compile_patterns–Port of
_compilePatterns(rule.ts:487-572): compiles the raw pattern -
create_capture_rule–Port of
createCaptureRule(rule.ts:383-387): registers the capture -
get_compiled_rule_id–Port of
getCompiledRuleId(rule.ts:389-447): the dispatch on the raw
compile_captures
staticmethod
¶
compile_captures(
captures: IRawRuleDict | None,
helper: IRuleFactoryHelper,
repository: IRawRepositoryDict,
) -> list[CaptureRule | None]
Port of _compileCaptures (rule.ts:449-485): builds the
(CaptureRule | None)[] indexed by capture group number 1..n.
compile_patterns
staticmethod
¶
compile_patterns(
patterns: list[IRawRuleDict] | None,
helper: IRuleFactoryHelper,
repository: IRawRepositoryDict,
) -> CompilePatternsResult
Port of _compilePatterns (rule.ts:487-572): compiles the raw pattern
list to rule ids, preserving order, resolving include entries and
dropping rules whose include chain resolves to nothing instantiated.
create_capture_rule
staticmethod
¶
create_capture_rule(
helper: IRuleFactoryHelper,
location: ILocation,
name: str | None,
contentName: str | None,
retokenizeCapturedWithRuleId: int,
) -> CaptureRule
Port of createCaptureRule (rule.ts:383-387): registers the capture
rule under a fresh id.
get_compiled_rule_id
staticmethod
¶
get_compiled_rule_id(
desc: IRawRuleDict,
helper: IRuleFactoryHelper,
repository: IRawRepositoryDict,
) -> int
Port of getCompiledRuleId (rule.ts:389-447): the dispatch on the raw
rule's shape, memoized on the rule dict via its internal id field.
Note on recursion: upstream writes desc.id = id inside the factory
callback, which is invoked synchronously by registerRule — so the id is
stored before any nested getCompiledRuleId runs, exactly as here.
RuleRegistry
¶
RuleRegistry(onig_lib: Any = None)
flowchart TD
textmate_grammar.engine.rules.RuleRegistry[RuleRegistry]
textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]
textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.rules.RuleRegistry
textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
click textmate_grammar.engine.rules.RuleRegistry href "" "textmate_grammar.engine.rules.RuleRegistry"
click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
Port of the rule-id <-> Rule bookkeeping that lives in Grammar
(grammar.ts:100, 127-128, 245-254): monotonically increasing numeric ids
starting at 1, with the rule objects stored under their id. Serves as the
concrete :class:IRuleFactoryHelper for :class:RuleFactory users and, later
(task 3), as the registry component of Grammar.
Note: upstream v9.3.2 has no standalone RuleRegistry class and no scope-name
interning; the class here is simply the extracted Grammar behaviour.
Methods:
-
create_onig_scanner–Port of
Grammar.createOnigScanner(grammar.ts:177-179). -
create_onig_string–Port of
Grammar.createOnigString(grammar.ts:180-181). -
get_external_grammar–Port of
Grammar.getExternalGrammar(grammar.ts:256-274): scope-name -
get_rule–Port of
Grammar.getRule(grammar.ts:252-254). May return the -
register_rule–Port of
Grammar.registerRule(grammar.ts:245-250).
create_onig_scanner
¶
Port of Grammar.createOnigScanner (grammar.ts:177-179).
create_onig_string
¶
Port of Grammar.createOnigString (grammar.ts:180-181).
get_external_grammar
¶
get_external_grammar(
scope_name: str, repository: IRawRepositoryDict | None
) -> IRawGrammarDict | None
Port of Grammar.getExternalGrammar (grammar.ts:256-274): scope-name
includes resolve through the grammar repository; without a registry layer
behind this helper there is nothing to resolve (task 4b provides it).
get_rule
¶
Port of Grammar.getRule (grammar.ts:252-254). May return the
(python) None of a not-yet-registered rule while its factory is still
running — upstream yields undefined there, and the only reader that
can act on it treats non-None like undefined (see
RuleFactory.compile_patterns).
create_rule
¶
create_rule(
desc: IRawRuleDict,
helper: IRuleFactoryHelper,
repository: IRawRepositoryDict,
) -> int
Mirror of RuleFactory.getCompiledRuleId (task-card spelling); prefer
:meth:RuleFactory.get_compiled_rule_id.
init_grammar
¶
init_grammar(
grammar: IRawGrammarDict,
base: IRawRuleDict | None = None,
) -> IRawGrammarDict
Port of initGrammar (grammar.ts:410-421): deep-copies the raw grammar
and installs the $self and $base repository entries the include
resolution relies on ($self = the top-level patterns, $base = the
caller-provided base grammar's self entry or the grammar's own).
parse_include
¶
parse_include(include: str) -> IncludeReference
Port of parseInclude (grammarDependencies.ts:278-295): $base /
$self, #rule (relative), scope (top level), scope#rule
(top-level repository reference). Note: this is the shared include parser; the
full grammarDependency machinery (task 4b) should import it from here.
tokenize
¶
The vscode-textmate v9.3.2 state model and line tokenization engine.
The main loop follows grammar/tokenizeString.ts. Scope names and state are
kept as immutable parent-linked structures, while enter/anchor positions are
line-local mutable bookkeeping exactly as in StateStackImpl.
Classes:
-
AttributedScopeStack–Scope path node plus vscode-textmate's inherited token/style attributes.
-
BalancedBracketSelectors–Balanced-bracket selector matcher for the optional binary-token API.
-
Grammar–A loaded grammar and its vscode-textmate line tokenizer.
-
LineTokens–Minimal
LineTokensimplementation for scope-array tokens. -
ScopeStack–The
theme.tslinked scope path, in root-first observable order. -
StateStackImpl–Port of grammar.ts
StateStackImpl(including line-scoped positions).
Functions:
-
create_grammar–Construct a grammar;
external_grammaris the registry integration seam.
AttributedScopeStack
dataclass
¶
AttributedScopeStack(
parent: AttributedScopeStack | None,
scope_path: ScopeStack,
token_attributes: int = 0,
font_attributes: Any = None,
style_attributes: Any = None,
)
Scope path node plus vscode-textmate's inherited token/style attributes.
BalancedBracketSelectors
¶
Balanced-bracket selector matcher for the optional binary-token API.
Grammar
¶
Grammar(
scope_name: str,
grammar: Mapping[str, Any],
initial_language: int = 1,
embedded_languages: Mapping[str, int] | None = None,
token_types: Mapping[str, int] | None = None,
balanced_bracket_selectors: BalancedBracketSelectors
| None = None,
grammar_repository: Any = None,
onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
*,
external_grammar: Callable[
[str], Mapping[str, Any] | None
]
| None = None,
)
flowchart TD
textmate_grammar.engine.tokenize.Grammar[Grammar]
textmate_grammar.engine.rules.RuleRegistry[RuleRegistry]
textmate_grammar.engine.rules.IRuleFactoryHelper[IRuleFactoryHelper]
textmate_grammar.engine.rules.IRuleRegistry[IRuleRegistry]
textmate_grammar.engine.rules.IGrammarRegistry[IGrammarRegistry]
textmate_grammar.engine.rules.RuleRegistry --> textmate_grammar.engine.tokenize.Grammar
textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.rules.RuleRegistry
textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IRuleFactoryHelper --> textmate_grammar.engine.tokenize.Grammar
textmate_grammar.engine.rules.IRuleRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
textmate_grammar.engine.rules.IGrammarRegistry --> textmate_grammar.engine.rules.IRuleFactoryHelper
click textmate_grammar.engine.tokenize.Grammar href "" "textmate_grammar.engine.tokenize.Grammar"
click textmate_grammar.engine.rules.RuleRegistry href "" "textmate_grammar.engine.rules.RuleRegistry"
click textmate_grammar.engine.rules.IRuleFactoryHelper href "" "textmate_grammar.engine.rules.IRuleFactoryHelper"
click textmate_grammar.engine.rules.IRuleRegistry href "" "textmate_grammar.engine.rules.IRuleRegistry"
click textmate_grammar.engine.rules.IGrammarRegistry href "" "textmate_grammar.engine.rules.IGrammarRegistry"
A loaded grammar and its vscode-textmate line tokenizer.
Methods:
-
create_onig_scanner–Port of
Grammar.createOnigScanner(grammar.ts:177-179). -
create_onig_string–Port of
Grammar.createOnigString(grammar.ts:180-181). -
get_rule–Port of
Grammar.getRule(grammar.ts:252-254). May return the -
register_rule–Port of
Grammar.registerRule(grammar.ts:245-250).
create_onig_scanner
¶
Port of Grammar.createOnigScanner (grammar.ts:177-179).
create_onig_string
¶
Port of Grammar.createOnigString (grammar.ts:180-181).
get_rule
¶
Port of Grammar.getRule (grammar.ts:252-254). May return the
(python) None of a not-yet-registered rule while its factory is still
running — upstream yields undefined there, and the only reader that
can act on it treats non-None like undefined (see
RuleFactory.compile_patterns).
ScopeStack
¶
ScopeStack(parent: ScopeStack | None, scope_name: str)
The theme.ts linked scope path, in root-first observable order.
StateStackImpl
¶
StateStackImpl(
parent: StateStackImpl | None,
rule_id: int,
enter_pos: int,
anchor_pos: int,
begin_rule_captured_eol: bool,
end_rule: str | None,
name_scopes_list: AttributedScopeStack | None,
content_name_scopes_list: AttributedScopeStack | None,
)
Port of grammar.ts StateStackImpl (including line-scoped positions).
create_grammar
¶
create_grammar(
scope_name: str,
grammar: Mapping[str, Any],
initial_language: int = 1,
embedded_languages: Mapping[str, int] | None = None,
token_types: Mapping[str, int] | None = None,
balanced_bracket_selectors: BalancedBracketSelectors
| None = None,
grammar_repository: Any = None,
onig_lib: IOnigLib = DEFAULT_ONIG_LIB,
*,
external_grammar: Callable[
[str], Mapping[str, Any] | None
]
| None = None,
) -> Grammar
Construct a grammar; external_grammar is the registry integration seam.
parsers
¶
Modules:
-
markdown–Small Markdown convenience wrapper around the public grammar engine.
-
matlab–Small MATLAB convenience wrapper around the public grammar engine.
markdown
¶
Small Markdown convenience wrapper around the public grammar engine.
Functions:
-
markdown_grammar–Return the module-cached Markdown grammar.
-
tokenize_markdown–Tokenize a Markdown source path or text, returning tokens grouped by line.
matlab
¶
Small MATLAB convenience wrapper around the public grammar engine.
Functions:
-
matlab_grammar–Return the module-cached MATLAB grammar.
-
tokenize_matlab–Tokenize a MATLAB source path or text, returning tokens grouped by line.