Consolidate the script codepoint table in _policy; classify 〆 as Han

docs/locales.rst

@@ -280,7 +280,11 @@ by ``tests/v2/test_locales.py``:
 #. Declare a module-level ``DEVIATES(name)`` predicate: given a name
    string, return whether *this pack alone* might parse it differently
    from the default parser. Over-declaring is safe; under-declaring is
-   not — when in doubt, ``DEVIATES`` should say yes.
+   not — when in doubt, ``DEVIATES`` should say yes. A pack whose
+   scope is a *script* builds the predicate with ``_script_matcher``
+   from ``nameparser/_policy.py``, the way ``zh`` and ``ja`` do —
+   never by compiling its own character ranges: the factory's
+   docstring explains how the pack-contract test enforces this.
 #. Add a rotator list to ``tests/v2/test_locales.py``. Every pack needs
    one, but what it has to contain follows from how the pack declares
    its scope. A pack declaring by *marker regex* (``ru``, ``tr_az``)

docs/release_log.rst

@@ -22,6 +22,7 @@ Release Log
     - Fix names written wholly in Han or Hangul parsing given-first: native-script CJK now reads family-first by default, through the new ``Policy.script_orders`` table, so ``"毛 泽东"`` gives family ``毛`` where 1.x gave family ``泽东``. A name that is a single unspaced token moves the same way — ``"毛泽东"`` and ``"山田太郎"`` now land in ``family``/``last`` where 1.x put them in ``given``/``first``, with the string itself untouched, which makes it the easiest form of this change to miss — a lone token renders the same whichever field holds it, whereas the spaced form does show up in the output (``str(HumanName("毛 泽东"))`` is now ``"泽东 毛"``). No language detection is involved: Chinese and Japanese both write the family name first in native script, so the script settles the order without anyone having to know which language it is. Latin-script and mixed-script names are never affected, and an explicit comma still wins. **Default-on: changes parse output for wholly-CJK names**, through ``HumanName`` as well as the 2.0 API (closes #271)
     - Fix unspaced Korean names not splitting: the census surname list now ships as default vocabulary (``Lexicon.surnames``) with hangul segmentation on by default (``Policy.segment_scripts``), so ``"김민준"`` parses family ``김``, given ``민준`` where 1.x returned the whole string as ``first``. Rendering follows the split, so ``str(HumanName("김민준"))`` is now ``"민준 김"`` where 1.x echoed the input back unchanged. Nothing but Korean is written in hangul and its surnames are a closed census set, which is what makes the split safe as a default rather than a pack. **Default-on**, and it reaches ``HumanName`` too. ``Policy(segment_scripts=())`` turns the split off; ``Policy(script_orders={})`` separately restores the positional reading; clearing both restores 2.0 behavior exactly (#271)
     - Fix Japanese names carrying kana parsing given-first: the family-first rule above extends to any name whose characters stay within kanji and kana while carrying at least one kana character, so ``"高橋 みなみ"`` gives family ``高橋`` and ``"山田 エミ"`` family ``山田`` where 1.x read both the other way round, and a lone such token (``"高橋みなみ"``, ``"みなみ"``) lands in ``family``/``last`` where 1.x put it in ``given``/``first``. The reasoning is the one hangul already uses: hiragana never transcribes a foreign name, and a transcription is kana ALONE, so kanji-plus-kana is a Japanese person's name written in Japanese order. A name written **wholly in katakana** is deliberately excluded and stays positional — it is predominantly a transcribed foreign name (``"マイケル ジャクソン"``) already in given-first order. **Default-on: changes parse output for kana-bearing Japanese names**, through ``HumanName`` as well as the 2.0 API, and the spaced forms change what the name renders as (``str(HumanName("高橋 みなみ"))`` is now ``"みなみ 高橋"``). ``Policy(script_orders={})`` clears this entry along with the Han and Hangul ones (#272)
+    - Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303)
     - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is deliberately NOT included: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``) (#272)
     - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272)
 

nameparser/_pipeline/_vocab.py

@@ -10,10 +10,11 @@
 
 import re
 import unicodedata
-from collections.abc import Iterable
+from collections.abc import Callable, Iterable
 
 from nameparser._lexicon import Lexicon, _normalize
-from nameparser._policy import Script
+from nameparser._policy import (Script, _JA_SCRIPTS, _SCRIPT_RANGES,
+                                _script_matcher)
 
 # Ported verbatim from v1 (nameparser/config/regexes.py "initial") minus
 # its empty-string alternative -- WorkToken text is never empty. Kept in
@@ -31,96 +32,24 @@
 PH = re.compile(r"^ph\.?$", re.IGNORECASE)
 D = re.compile(r"^d\.?$", re.IGNORECASE)
 
-# Codepoint ranges per Script (#271). This integer table is the single
-# source of truth for what a script covers; _SCRIPT_PATTERNS below
-# DERIVES the match engine from it. (The sweep here was first written
+# The codepoint table lives in _policy beside Script -- one copy
+# importable from the pipeline and the locale packs alike; everything
+# here DERIVES from it. (single_script's sweep was first written
 # per-char on the _EMOJI_RANGES precedent in _tokenize.py, on the
 # theory that a range test needs no regex; measured at token scale the
-# compiled regex wins by 3-9x, and by 89x on long tokens.)
-# HAN: the ideographic iteration mark U+3005, the URO plus Extension
-# A, the compatibility block, and the supplementary-plane block
-# (Ext B-I + CJK Compat Ideographs Supplement, 0x20000-0x323AF) --
-# rare surnames are the biggest real source of supplementary-plane
-# hanzi in personal names (e.g. 𠮷田's 𠮷, U+20BB7), so leaving them
-# out silently mis-orders those names; unassigned gaps inside the span
-# are harmless, since no real name contains an unassigned codepoint.
-# U+3005 々 is the block-vs-Script case, running the OPPOSITE way to
-# U+30FB below: 々 already IS Script=Han under UAX #24 (Scripts.txt
-# reads `3005 ; Han`), but it sits in CJK Symbols and Punctuation,
-# outside every CJK ideograph block this table spans -- so the
-# singleton entry is what a BLOCK table needs to reach a character the
-# Script property would have classified correctly for free. It earns
-# the reach: 々 repeats the preceding kanji and appears only inside
-# Han-written names -- 佐々木 (Sasaki, a top-20 Japanese surname),
-# 野々村, 奈々. Omitting it made 佐々木 a mixed-script token: the name
-# reversed and never gated into segmentation.
-# HANGUL: precomposed syllables only -- modern Korean
-# text never writes names as bare jamo.
-# HIRAGANA/KATAKANA (#272): the two kana blocks, each in full. There
-# IS a supplementary-plane kana repertoire (Kana Supplement, Kana
-# Extended-A/B, Small Kana Extension, U+1AFF0-U+1B16F, a few hundred
-# assigned codepoints -- no exact count here, it moves with the
-# Unicode version) but none of it is WORTH chasing the way Han's astral
-# block is: those codepoints are hentaigana and other archaic/
-# phonetic-extension forms no modern Japanese name uses, unlike
-# supplementary Han, which real surnames genuinely need. The Katakana
-# Phonetic Extensions block (U+31F0-U+31FF, 16 small katakana for Ainu
-# transcription) is excluded for the same reason -- no modern Japanese
-# personal name uses them. Halfwidth kana (U+FF65-U+FF9F, including
-# the voiced/semi-voiced sound marks U+FF9E/U+FF9F) is likewise
-# deliberately excluded -- legacy bank/CSV data uses it, but it is a
-# separate normalization problem; Task 2b's separator handling only
-# touches the halfwidth DOT (U+FF65), not the rest of that block.
-# This table classifies by Unicode BLOCK, not the UAX #24 Script
-# property: U+30A0, U+30FB (the middle dot), and U+30FC (the
-# prolonged sound mark) all carry Script=Common under UAX #24, and the
-# four kana voicing marks U+3099-U+309C split two and two -- U+3099
-# and U+309A are the COMBINING forms (Script=Inherited), U+309B and
-# U+309C the spacing ones (Script=Common) -- yet every one of them is
-# needed here, and block membership, not the Script property, is what
-# puts them in range. The katakana block's upper end (U+30FF) takes in
-# the middle dot U+30FB, kept rather than carved out for a smaller
-# reason than it looks: tokenize (#272 Task 2b) turns U+30FB into a
-# token separator, so no real parse shows this classifier a string
-# containing one. It is kept so that a DIRECT whole-string call --
-# effective_script("マイケル・ジャクソン"), which the unit tests make --
-# still classifies instead of returning None. The ranges below must stay
-# mutually disjoint: single_script returns the FIRST covering entry
-# (dict iteration order), so an overlapping future script would make
-# the result order-dependent instead of well-defined.
-_SCRIPT_RANGES: dict[Script, tuple[tuple[int, int], ...]] = {
-    Script.HAN: ((0x3005, 0x3005), (0x3400, 0x4DBF), (0x4E00, 0x9FFF),
-                 (0xF900, 0xFAFF), (0x20000, 0x323AF)),
-    Script.HANGUL: ((0xAC00, 0xD7A3),),
-    Script.HIRAGANA: ((0x3040, 0x309F),),
-    Script.KATAKANA: ((0x30A0, 0x30FF),),
+# compiled regex wins by 3-9x, and by roughly 70x on long tokens.)
+# Derived, never hand-written -- even the class construction goes
+# through the shared factory: one wholly-of predicate per script, in
+# the table's key order -- the FIRST-covering-entry rule _classify
+# documents.
+_SCRIPT_MATCHERS: dict[Script, Callable[[str], bool]] = {
+    script: _script_matcher(script, whole=True)
+    for script in _SCRIPT_RANGES
 }
 
-# Derived, never hand-written: one character class per script, in the
-# table's own key order (so the FIRST-covering-entry rule above still
-# describes what single_script does).
-_SCRIPT_PATTERNS: dict[Script, re.Pattern[str]] = {
-    script: re.compile(
-        "[" + "".join(f"\\U{lo:08x}-\\U{hi:08x}" for lo, hi in ranges)
-        + "]+")
-    for script, ranges in _SCRIPT_RANGES.items()
-}
-
-#: The Japanese repertoire: the union effective_script's kana license
-#: quantifies over -- HAN, HIRAGANA, KATAKANA (HANGUL simply omitted).
-#: A frozenset, not the tuple this started as: resolve_script_set
-#: below is the "later task that needs membership" the tuple's
-#: original comment anticipated. Membership doesn't care about order,
-#: and the pattern built below doesn't either (a regex character
-#: class matches the same set regardless of the order its ranges are
-#: written in).
-_JA_SCRIPTS = frozenset({Script.HAN, Script.HIRAGANA, Script.KATAKANA})
-_JA_PATTERN = re.compile(
-    "["
-    + "".join(f"\\U{lo:08x}-\\U{hi:08x}"
-              for s in _JA_SCRIPTS
-              for lo, hi in _SCRIPT_RANGES[s])
-    + "]+")
+# The whole-token matcher over _policy's _JA_SCRIPTS union, backing
+# effective_script's kana license.
+_wholly_ja = _script_matcher(*_JA_SCRIPTS, whole=True)
 
 
 def is_initial(text: str) -> bool:
@@ -250,11 +179,11 @@ def _normalized_for_script(text: str) -> str | None:
 
 
 def _classify(normalized: str) -> Script | None:
-    """The FIRST _SCRIPT_PATTERNS entry covering all of `normalized`
+    """The FIRST _SCRIPT_MATCHERS entry covering all of `normalized`
     (already NFC, via _normalized_for_script), else None. Shared by
     both public classifiers so each of them normalizes exactly once."""
-    for script, pattern in _SCRIPT_PATTERNS.items():
-        if pattern.fullmatch(normalized):
+    for script, matcher in _SCRIPT_MATCHERS.items():
+        if matcher(normalized):
             return script
     return None
 
@@ -282,7 +211,7 @@ def effective_script(text: str) -> Script | None:
     HIRAGANA carrier entry. Pure-katakana stays KATAKANA
     (single_script's answer): a lone katakana token is predominantly a
     transcribed foreign name, so nothing defaults on it."""
-    # None for both shapes _JA_PATTERN could never match anyway (empty
+    # None for both shapes _wholly_ja could never match anyway (empty
     # text, or all-ASCII text): real work, not a leftover "if text"
     # guard, since the ASCII case is one a bare emptiness check would
     # let through. The single normalized copy then serves both the
@@ -293,7 +222,7 @@ def effective_script(text: str) -> Script | None:
     script = _classify(normalized)
     if script is not None:
         return script
-    if _JA_PATTERN.fullmatch(normalized):
+    if _wholly_ja(normalized):
         return Script.HIRAGANA
     return None
 
@@ -322,6 +251,6 @@ def resolve_script_set(scripts: Iterable[Script]) -> Script | None:
     found = frozenset(scripts)
     if len(found) <= 1:
         return next(iter(found), None)
-    if found <= _JA_SCRIPTS:
+    if found.issubset(_JA_SCRIPTS):
         return Script.HIRAGANA
     return None