You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
tools/differential/ is the pre-release gate that diffs the v2 facade against pinned 1.4-from-PyPI, and docs/release_log.rst's Behavior Changes section names it as its source. It is currently structurally blind to 2.1's headline change: neither corpus contains a single CJK character.
The blindness is by construction, not accident:
corpus.jsonl regenerates from v1's own test banks at an immutable ref, and v1's authors had no reason to test CJK input.
Consequence: the fix(#271) rule in expected_changes.toml is pinned against _vocab._SCRIPT_RANGES by test_regex_sync.py and was verified against a throwaway synthetic corpus during #294, but classifies nothing in a real run. The behavioral guarantee currently lives in the 17 fix(#271) rows in tests/v2/cases.py.
The fix needs a provenance decision before any code, because the README deliberately forbids hand-authored corpora ("a corpus you have to ask for by name is a corpus that stops being run" cuts the other way here — compare.py globs corpus*.jsonl, so a new file would run automatically):
Sanction a third file (corpus_cjk.jsonl) with its own documented provenance — e.g. generated from the fix(#271) case-table rows, which are themselves reviewed. Deterministic coverage; adds a third provenance class to the README's table.
Either way, the acceptance test is the same: a real compare.py run in which the fix(#271) rule claims at least one name, so the rule can never again be dead-on-arrival without the run saying so (the failure mode #294 caught only by synthetic probe).
tools/differential/is the pre-release gate that diffs the v2 facade against pinned 1.4-from-PyPI, anddocs/release_log.rst's Behavior Changes section names it as its source. It is currently structurally blind to 2.1's headline change: neither corpus contains a single CJK character.The blindness is by construction, not accident:
corpus.jsonlregenerates from v1's own test banks at an immutable ref, and v1's authors had no reason to test CJK input.build_issues_corpus.pyrequires an internal space in a harvested name-like string — and unspaced CJK names, the exact shape Unspaced Chinese/Korean names: surnames constants + longest-match segmentation #271 classifies, never have one. (Verified during Parse native-script CJK names by default and add the ZH pack (#271) #294: regenerating picked up CJK-adjacent issues' romanized examples only.)Consequence: the
fix(#271)rule inexpected_changes.tomlis pinned against_vocab._SCRIPT_RANGESbytest_regex_sync.pyand was verified against a throwaway synthetic corpus during #294, but classifies nothing in a real run. The behavioral guarantee currently lives in the 17fix(#271)rows intests/v2/cases.py.The fix needs a provenance decision before any code, because the README deliberately forbids hand-authored corpora ("a corpus you have to ask for by name is a corpus that stops being run" cuts the other way here —
compare.pyglobscorpus*.jsonl, so a new file would run automatically):corpus_issues.jsonl. Keeps the harvested-provenance story; coverage depends on what the tracker happens to contain (Does not work for chinese name #83, Unspaced Chinese/Korean names: surnames constants + longest-match segmentation #271, and Japanese name segmentation via optional dependency (nameparser[ja]) #272 all carry native-script examples today).corpus_cjk.jsonl) with its own documented provenance — e.g. generated from thefix(#271)case-table rows, which are themselves reviewed. Deterministic coverage; adds a third provenance class to the README's table.Either way, the acceptance test is the same: a real
compare.pyrun in which thefix(#271)rule claims at least one name, so the rule can never again be dead-on-arrival without the run saying so (the failure mode #294 caught only by synthetic probe).