Consolidate the script codepoint table in _policy; classify 〆 as Han - #303
Merged
Conversation
The codepoint table is data about Script -- which spans each enum member covers -- so it belongs beside the enum in the config layer, not in the pipeline. With one copy readable from below the layering boundary (packs may import _policy, never the pipeline), the hand-copied spans in the zh/ja locale packs can die in the following commits instead of being kept in sync by hand. This deliberately reverses the #271 placement decision that put the table in _pipeline/_vocab.py: that placement is what forced each pack to hand-copy its spans in the first place. _vocab now imports the table and still derives its per-script patterns from it; behavior is unchanged, and only test imports retarget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A factory beside _SCRIPT_RANGES that compiles one character class from the union of the named scripts' spans and returns a named closure predicate (an empty union raises ValueError rather than a cryptic pattern error). Two modes: whole=False answers CONTAINS-any -- DEVIATES' contract, where over-declaring is the gate's safe direction; whole=True answers non-empty-and-WHOLLY -- the ja adapter's repertoire guard. The compiled re.Pattern lives inside the closure, and compilation lives in _policy rather than in a pack-local closure, deliberately: tests/v2/test_locales.py classifies any pack module holding a module-level re.Pattern as a marker pack requiring rotator branch coverage, and its registry gate fails a pack that merely IMPORTS re without exposing one -- so packs must not import re at all. Predicates built here keep the zh/ja packs invisible to that sweep by construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The hand-copied Han spans and their sync test die because the table is now importable from _policy. DEVIATES keeps its documented def-with-docstring shape (the pack-authoring contract in docs/locales.rst) and delegates to the factory closure; the per-character loop is gone (the compiled character class was measured 16-30x faster at DEVIATES' scan scale). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The three-script hand-copy (_JA_RANGES) and both per-char loops die: the pack now names its scripts once (_JA_SCRIPTS) and lets _policy's _script_matcher compile the two predicates from the shared codepoint table, so there are no spans here to drift out of sync. DEVIATES keeps the documented def shape, delegating to the contains-any matcher; the adapter's guard becomes one whole-token predicate call with identical semantics (empty text declines one check earlier, same None outcome). The sync pin in tests/v2/test_locales.py is replaced by floor/ceiling declaration assertions in test_ja_pack_contents. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The two hand-written class-construction expressions were semantically identical to factory calls: _SCRIPT_PATTERNS was per-script wholly-of in the table's key order, _JA_PATTERN wholly-of over the three-script union. Folding them into _script_matcher(whole=True) deletes the last derivation duplicates of the factory's character-class expression and the _JA_PATTERN roster row in test_regex_sync (neither replacement is a module-level re.Pattern, so the completeness sweep no longer sees them). This is a review-driven addition beyond the original spec's "vocab matchers stay unchanged" non-goal, taken because the shared factory made that non-goal's rationale moot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_MARKER_REGEX_PACKS classifies any pack holding a module-level re.Pattern as a marker pack whose regex branches the rotator sweep must cover. Since zh and ja now derive their DEVIATES predicates through _script_matcher, that boundary is load-bearing: the factory's closure is what hides the compiled class and keeps range-declaring packs out of a branch-coverage sweep a codepoint-range predicate cannot satisfy. Nothing pinned it -- a pack author who "simplifies" a closure back to a module-level compiled pattern would silently flip zh or ja into the sweep. The new test converts that failure from silent to loud, and the comment above the derivation now names both packs (it predated ja's switch and named only zh). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one behavior change of this branch: the HAN entry's opening span widens from the 々 singleton (0x3005, 0x3005) to (0x3005, 0x3006), taking in 〆, the shime mark. Unlike 々 -- Script=Han under UAX #24, merely outside every CJK ideograph block -- 〆 carries Script=Common, so this is the table deliberately reaching PAST the Script property, not around a block boundary. It earns the reach: 〆 functions solely as a component of Japanese surnames (〆木 Shimeki, 〆谷 Shimetani, 〆野) and appears in no other script's names. Before the change 〆木 was a mixed-script token: unlicensed, positional, given-first -- the name reversed and never gated into segmentation. The widening also makes 〆木太郎 wholly Japanese to the ja pack's adapter, which would have handed it to namedivider -- whose 0.4.x rule path cuts every attested 〆 surname at offset 1 (〆 | 木太郎, score 1.0, algorithm='rule'; measured): a wrong family arriving with NO SEGMENTATION report, since 1.0 clears the floor. The adapter now declines 〆-bearing tokens before the divider is consulted (pinned stub-based in tests/v2/test_locales.py, no extra needed): the family-first ORDER fix stands regardless, and division can return when a divider knows the mark. The differential toml's character class is a hand copy of the table, and its sync guard tripped in between exactly as designed: test_differential_cjk_rule_matches_the_script_ranges failed with "expected_changes.toml's CJK name_regex declares [(12293, 12293), ...]; _SCRIPT_RANGES' BMP spans are [(12293, 12294), ...]" until the CJK rule's 々-々 was widened to 々-〆 to match. Differential harness re-run by hand: 654 corpus names, 18 intentional diffs, 0 unexplained, exit 0 -- neither corpus contains a 〆 name, so totals are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closing sweep for the _SCRIPT_RANGES move to _policy.py. Two stale
spots, both corrected:
- nameparser/_policy.py: _script_matcher's docstring claimed the
closure keeps derived matchers out of test_regex_sync.py's hand-copy
completeness sweep as if that sweep scanned everywhere; it scans a
fixed set of pipeline modules (_assign, _post_rules, _render,
_tokenize, _vocab), which covers _vocab but never _policy or the
packs. Reworded to claim only what holds: the closure spares
_vocab's derived matchers a declaration row in that sweep.
- docs/locales.rst: the contributing contract told authors to declare
DEVIATES but never said how a range-declaring pack builds one, while
test_locales.py now reads any module-level re.Pattern as a marker
regex needing branch coverage and fails a pack importing re without
one. Added the mechanism: build the predicate with _policy's
_script_matcher, the way zh and ja do, never by compiling private
character ranges.
Verified rather than edited: AGENTS.md's test_regex_sync.py line
("wherever the copy lives -- including copies outside the package")
stays true because the differential toml copy in tools/ remains and is
still pinned; its layering line already lists _policy among allowed
pack imports. Acceptance greps: 0x4E00 appears in the package only in
_policy.py's table; the dead symbols (_HAN_RANGES, _JA_RANGES,
_in_repertoire, _SCRIPT_PATTERNS, _JA_PATTERN) are gone from
nameparser/, tests/, and tools/; no pack references _vocab.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #303 +/- ##
=======================================
Coverage 98.43% 98.43%
=======================================
Files 41 41
Lines 2739 2750 +11
=======================================
+ Hits 2696 2707 +11
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Comment corrections:
- _policy.py: scope the 〆 justification to personal names -- the mark
is a general-purpose Japanese symbol (〆切, the envelope closing
mark); only its within-names use is surname-only
- _policy.py: "a pack must not import re" -> "a range-declaring pack"
(marker packs like ru/tr_az import re legitimately, by exposing
module-level patterns)
- _policy.py: the regex-sync sweep scans the pipeline modules PLUS
_render, and only private (underscore-prefixed) module-level
patterns -- say so
- _policy.py: replace the hand-copy PR-history parenthetical with the
durable constraint (the table lives here because packs must not
import the pipeline)
- locales/ja.py: reflow two ragged comment wraps (wording unchanged)
- locales/ja.py: acknowledge the blanket 〆 decline's deliberate
breadth -- mid-token 〆 (山〆太郎) measures 0.375 on the
kanji_feature path and WOULD have carried a SEGMENTATION report;
swept in anyway because decline is the safe direction
New pins:
- test_locales.py guard-stack stub test: assert seg("Yamada太郎") is
None -- kills a mutation dropping whole=True from _wholly_japanese,
which survived the whole suite
- test_locales.py pack-contents tests: _zh/_ja.DEVIATES("𠮷田") --
pins the astral extent of the pack predicates, which a BMP-only
hand-rolled replacement would otherwise survive
- cases.py ja_shime_with_kana_given ("〆木 ひろ") -- the kana license
composing with the widened Han span, previously untested together
- test_policy.py test_every_script_member_has_a_range_entry -- a
future Script member without a _SCRIPT_RANGES entry is accepted by
Policy validation yet never classified; this turns that silent
no-op into a failure
- test_locales.py test_namedivider_still_miscuts_shime_names -- the
self-expiring canary behind the adapter's 〆 decline; when
namedivider learns the mark this fails and the decline gets
revisited instead of lasting forever
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… prose
* Hoist _JA_SCRIPTS into _policy.py beside _SCRIPT_RANGES: the union
was defined twice (_pipeline/_vocab.py frozenset, locales/ja.py
tuple) with nothing pinning their membership equality -- the same
drift class this branch already deleted at span level. One tuple
now serves the kana license, the ja pack's DEVIATES, and the
adapter's repertoire guard; the katakana rationale moves onto the
ja predicates it explains, and _vocab's naming-defense comment
(guarding a constraint that never existed) goes away.
* Merge _script_matcher's twin closures: one compiled [cls]+ pattern
with the bound method chosen by `whole` -- search over [cls]+ is
exactly contains-any, so the second pattern and closure bought
nothing.
* Reorder ja adapter guards: the O(1) length check now leads the
chain, so routine one-char tokens from spaced input ("林 太郎") no
longer pay the regex fullmatch and the 〆 scan (measured 4x) before
being declined. All guards return the same None, so behavior is
unchanged.
* Trim four prose duplications: the 〆 paragraph's copy of the 々
closer in _policy's table comment (plus an orphan line wrap), the
triple-stated ordering point on _SCRIPT_MATCHERS, locales.rst's
restatement of the marker-classification contract the factory
docstring already carries, and the twice-narrated namedivider
mis-cut measurement in test_locales.py (the authoritative account
stays at ja.py's decline comment).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Relocation refactor from the 2.1 CJK work's follow-up list, plus one small behavior fix that rides the same table entry.
The move.
_SCRIPT_RANGES— the codepoint table behind every script decision — moves from_pipeline/_vocab.pyto_policy.py, beside theScriptenum it describes. Layering bars packs from importing the pipeline, so the old placement had forced two hand-copies (zh._HAN_RANGES,ja._JA_RANGES), each with a sync test. All of that dissolves: packs and pipeline now read one table.The factory. A new
_policy._script_matcher(*scripts, whole=False)compiles a character class from the union of the named scripts' spans and returns a closure predicate (contains-any, or wholly-of withwhole=True). The compiled pattern living inside the closure is load-bearing: a module-levelre.Patternin a pack flips the marker-pack classification intest_locales.py, and a pack importingreat all trips the registry gate — a new pin asserts zh/ja stay out of that classification. The zh/ja DEVIATES gates and the ja adapter's repertoire guard now derive through the factory (the per-char loops they replace measured ~9× slower on short tokens, ~70× on long ones), as do_vocab's own script classifiers — a review-driven extension that deleted the last duplicate derivations.〆 (U+3006) classifies as Han — the one behavior change. 〆 opens real Japanese surnames (〆木, 〆谷, 〆野) but carries Script=Common under UAX #24, so this is the block table deliberately reaching past the Script property (々's singleton was the opposite case: Script=Han outside every ideograph block). Effect: 〆-surnames gain the East Asian family-first reading. The ja segmenter adapter declines 〆-bearing tokens — namedivider 0.4.x's rule path cuts every attested 〆 surname one character early at confidence 1.0, which would have produced a wrong family with no SEGMENTATION report; the decline is pinned by a stub test that runs without the extra. The differential harness's hand-copied character class widened in lockstep (its sync pin tripped mid-commit, as designed) — the one irreducible copy, since TOML can't import Python.
Test plan
parse("〆木 太郎")→ family 〆木, given 太郎;parse("〆木太郎")→ family whole🤖 Generated with Claude Code