Parse native-script CJK names by default and add the ZH pack (#271) - #294
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#271) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#271) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unspaced CJK names give tokenize no separator to find, so the new stage inserts the missing boundary by vocabulary: the first token written wholly in an activated script is matched longest-first against Lexicon.surnames and split into two sub-slices of the original. It runs after segment, on the comma doctrine that script-conditional behavior is ignored where a comma already decides the family: FAMILY_COMMA opts out whole, and the remaining structures split within the name segment only, remapping the segment runs past the insertion. Inert by default -- the default surname vocabulary is still empty. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Han ORDER is already default (script_orders reads a wholly-Han name
family-first); Han SEGMENTATION is the one behavior that must stay
opt-in, because a pure-Han string cannot say whether it is Chinese or
Japanese and a zh surname list corrupts Japanese names. locales.ZH is
that opt-in: PolicyPatch(segment_scripts={Script.HAN}) plus the
vocabulary to segment with -- 2020 census top-100 singles, their
traditional variant forms, and the Hundred Family Surnames compounds
still in use. It sets no order field at all, so applying it can only
ADD: Korean segmentation keeps working through a zh parser.
Two entries deviate from census rank order and say so in the data
comment: 阎 sits off-rank beside its reading-pair 闫, and 萧 is
appended because the mainland census records that surname under the
merged 肖 while both simplified spellings circulate.
DEVIATES declares any name containing a Han character. That
over-declares (a Han name needs an unspaced surname match to actually
change), which is the gate's safe direction.
Two deferred items ride along. script_segment's per-parse
max(map(len, surnames)) becomes a per-vocabulary lru_cache -- the pack
roughly triples the surname set for its users, and Lexicon is frozen
and slotted, so it cannot carry the cache itself. And the property
layer's vocabulary pool gains CJK entries, with _names_using offering
each drawn surname concatenated with a given name: script_segment
splits an unspaced token, so a space-joined name could never reach it
(measured: the stage fired on none of the 250 config-fuzz examples).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The _policies strategy hand-lists Policy's fields, so segment_scripts and script_orders -- 2.1's whole new configuration surface -- were the only ones nothing fuzzed. Draw both: script_orders from four sampled tables (the values are as restricted as name_order's), segment_scripts over the full Script enum, which is what lets a drawn hangul or Han surname actually reach the script_segment stage. AGENTS.md gains the scoped exception to the language-agnostic rule the amendment asks the implementing commit to record -- script-conditional behavior exactly where the script itself, not statistics about it, determines the convention -- plus surnames.py and maiden_markers.py in the config-module list, noted as the two data modules Constants has no attribute for. The cross-cutting test-module list was stale by six files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the East Asian sections the 2.1 behavior needs (usage.rst for what it does, customize.rst for turning each half off, locales.rst for why Korean needs no pack and Chinese does), and corrects the claims #271 made false: locales.rst said both shipped packs were policy-only and told contributors to keep them that way, concepts.rst said name_order and a comma were the only things deciding first-vs-last, and the RFC listed the Chinese/Korean work as still staged. The opt-out story is stated carefully because the obvious reading of it is wrong: script_orders={} restores the positional read but leaves the token split, so on a Korean name it moves the surname into "given" rather than undoing anything. Clearing both fields is what restores 2.0. Also fixes AGENTS.md's doctest note, which claimed Sphinx isolates local variables between blocks. It does not -- an example added here bound "name" and broke four passing examples further down usage.rst. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the expected_changes.toml rule for the two default changes, scoped to a CJK character class so it can only claim names the behavior can actually reach, and regenerates corpus_issues.jsonl (198 -> 200 names; the generator only ever adds). The rule goes FIRST, which turned out to be load-bearing. classify() takes the first matching rule and fix(suffix-routing) declares no name_regex at all, so with the new rule appended at the end -- where it started -- every CJK diff was claimed by suffix-routing under the wrong label and the new rule was dead on arrival. Caught by running the harness against a synthetic CJK corpus, since neither checked-in corpus contains a single CJK character: build_issues_corpus.py requires an internal space and unspaced names are the shape this classifies. The ordering rule is now written down in both the file header and the README. The character class is a hand copy of _vocab._SCRIPT_RANGES that no import can keep honest, so test_regex_sync.py pins it -- the module's first pin on a file outside the package, and its quietest copy, since the harness runs by hand rather than in CI. Han's astral block is excluded on both sides, deliberately: no corpus name reaches it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two claims about 1.4 in migrate.rst were wrong, both verified against
the pinned 1.4 worker rather than from memory:
- 1.4 routes a LONE token to first, universally, so HumanName("김민준")
gave first="김민준" and not last. The doc said last.
- "Unspaced Chinese is unaffected either way" was false. Segmentation
does not apply to it, but script_orders does: 毛泽东 and 山田太郎 now
land in last where 1.4 put them in first. Nothing splits, so the
string looks identical and only the field changed -- the quietest
form of this release's default-on change, and the doc had told
readers there was nothing to check. Now stated plainly, with the
Japanese case named, since a kanji name is affected exactly as a
Chinese one is.
The release log's first #271 bullet illustrated only the spaced case
and inherited the same blind spot; it now covers the lone-token flip
too. Its parse-output bullets also move under a Behavior Changes
heading, matching the 2.0.0 block's structure.
Also rewrites the locales.rst contributor bullet that opened "Sort the
vocabulary before shipping it" -- leftover phrasing from the bullet it
replaced, and actively contradicting the "do not alphabetize" rank-order
convention that zh.py and surnames.py both document in the data itself.
The bullet is about which LAYER the vocabulary belongs in, so it says
that instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SEGMENTATION fork example was false for the shipped vocabulary. 欧
is not in the zh surname list, so 欧阳明 has exactly one possible split
and emits nothing -- the stage tests only show otherwise because their
synthetic lexicon adds 欧 on purpose. Both prose copies now use 夏侯惇,
where 夏侯 wins and 夏 is genuinely listed too, so the example describes
a fork the shipped data can actually produce (verified: it emits, and
test_locales already pins that reading). The stage tests keep 欧阳明 --
their lexicon is synthetic by design and says so.
Also documents the rendered-string change for Korean names, which no
doc mentioned: str(HumanName("김민준")) was "김민준" in 1.4 and is
"민준 김" now, because segmentation inserts a boundary the default
format then writes given-first.
Writing that turned up an error in the release log entry from the last
commit, fixed here: the claim that the Han case is invisible in the
output string holds only for a LONE token. The spaced form reorders,
so str(HumanName("毛 泽东")) is "泽东 毛" where 1.4 echoed the input.
Both bullets now say which shape they mean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #294 +/- ##
==========================================
+ Coverage 98.19% 98.30% +0.11%
==========================================
Files 37 40 +3
Lines 2434 2594 +160
==========================================
+ Hits 2390 2550 +160
Misses 44 44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The four #271 additions all sat on paths every parse walks, and three of them paid for shapes measurement did not support. single_script's per-char all/any sweep was written on the _EMOJI_RANGES precedent, on the theory that a range test needs no regex. At token scale that is backwards: a compiled character class per script runs 5x faster on ordinary CJK tokens and 35x on long ones. The integer table stays the single source of truth -- the patterns are DERIVED from it, in its own key order, so the first-covering-entry rule still describes what single_script does. script_segment now returns early on a wholly-ASCII original. Spans index the original exactly, so an ASCII original has only ASCII tokens and no script's ranges can contain any of them: the stage was building a lexicon lookup and a generator per Latin parse to prove it. _effective_order joined each piece's tokens into a throwaway string and built a set of results before asking whether the set had one member. WorkToken text is never empty, so "the piece is wholly one script" is exactly "every token in it is" -- a per-token walk with early exit says the same thing and stops at the first token that settles it. _policy.py loses two near-duplicate wrap-and-rethrow blocks (the probe-vs-consume rule now lives once, on _require_iterable, which takes the field's own phrasing as a parameter), a vestigial tuple() materialization, and PolicyPatch.__post_init__'s 77-line inline script_orders block, which becomes a named helper that always materializes. Error texts are unchanged; the one behavior nuance is invisible from outside -- a re-iterable input with a malformed entry is now stored as a materialized tuple rather than the original container, and Policy quotes entries, never containers. Tests: the Korean-surname range check reads the shared table instead of a second hand-copy of 0xAC00/0xD7A3, one comment that promised a stage as future work is reworded now that it shipped, and the zh anti-dedupe note covers the case-table overlap it had grown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
expected_changes.toml's rules were tried in file order, so a rule narrowed by name_regex had to be written above every rule narrowed only by fields -- an invariant the file could only defend in prose, and one it had already lost once (the fix(#271) CJK rule was dead on arrival at the end of the file, its diffs claimed and mislabelled by fields-only fix(suffix-routing)). compare.py now sorts the rules into two tiers before matching, stably, so rules within a tier keep the order they were written in. That makes the ordering rule structural, and the three places that documented it by hand -- the toml header, the #271 rule's own "FIRST on purpose" paragraph, and the README's ordering section -- collapse to one line each. No exit-code change: sorting cannot change WHICH rules match, only which of several matching rules wins the label. The run is unchanged against the current corpora (654 names, 18 intentional diffs, 0 unexplained, exit 0) -- with one fields-only rule in the file and the one specific rule that competes with it already written above it, the sort is a no-op today. Verified it is not a no-op in general: with the #271 rule moved to the end of the file, '毛泽东' classifies as suffix-routing unsorted and as fix(#271) sorted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The only uncovered lines the branch added (codecov/patch): the wrong-arity, str-valued, and non-iterable per-pair deferrals -- each now pinned as constructible/hashable and quoted by Policy at apply. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lead with how the names are built (order, unspaced writing, closed surname sets), then the two automatic behaviors, then the boundaries -- replacing the casual 'come out right' / 'a step further' framing that assumed the reader already knew the conventions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
locales.rst: state the naming conventions before the two defaults and name the actual limitation (the script cannot license a Han split) instead of 'waits for the zh pack'. customize.rst: name the mechanism behind the opt-out trap (the split still runs; the positional default then assigns given-first) instead of 'read it twice'. usage.rst: drop the self-referential lead-in sentence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 29, 2026
derek73
added a commit
that referenced
this pull request
Jul 29, 2026
Task 1 of the #272 Japanese-support plan: classification only, no order/segmentation defaults change. Script gains HIRAGANA (U+3040- U+309F) and KATAKANA (U+30A0-U+30FF, which includes the prolonged mark and the U+30FB middle dot). _vocab.effective_script extends single_script with the kana license from the 2026-07-29 amendment: a mixed Han/hiragana/katakana token is Japanese and resolves to the HIRAGANA carrier entry; pure-katakana keeps single_script's own answer since nothing defaults on it. Updated two pre-existing pins that this table change broke: the single_script test asserting kana was unclassified (now KATAKANA), and test_regex_sync's #271 differential-rule sync check, which now scopes its expected spans to the HAN/HANGUL scripts #271 actually governs rather than every _SCRIPT_RANGES entry, since #272 grows the same table for an unrelated reason. Quality-review fix-first pass, folded into this same commit: both classifiers now run on an NFC-normalized copy of the text, never the raw input. NFD decomposes precomposed katakana onto a base character plus a combining voiced/semi-voiced sound mark (U+3099/U+309A), which sit in the HIRAGANA block rather than katakana's -- classifying raw NFD text could therefore see a pure-katakana token as Han-free but kana-mixed and wrongly hand it the kana license (HIRAGANA) instead of declining it (KATAKANA). Separately, NFD decomposes Hangul syllables onto bare jamo (U+1100-U+11FF), entirely outside the HANGUL range, so raw NFD Korean input missed the shipped family-first order rule altogether and silently fell back to the positional default -- a live gap in #294's shipped behavior, not merely a #272 nicety. Both directions are demonstrated in tests/v2/pipeline/test_vocab.py by simulating the pre-fix classifier against NFD input built with unicodedata.normalize() (never pasted as decomposed literals): the pre-fix classifier returns None for both an NFD katakana and an NFD hangul string, and would have returned HIRAGANA (wrongly) for the raw NFD katakana case under a naive license check. Matching (is_initial, suffix lookups, etc.) deliberately stays on raw text elsewhere in the module -- NFD only ever costs a match there, never a wrong one, so that asymmetry is safe and unchanged. The normalized copy is used for classification only; token text and rendered spans are exactly what the caller wrote, still NFD if that's what came in (see the new test_nfd_korean_input_still_reads_family_first in test_parser.py, which compares NFC-normalized field values since the rendered family/ given text itself stays NFD). Also folded in from the same review: reworded the "no supplementary- plane kana to chase" claim (false -- 311 assigned codepoints exist; none are worth chasing, since they're archaic/phonetic-extension forms no modern name uses); added the halfwidth-kana exclusion rationale and the block-vs-UAX#24 clause to the range-table comment; corrected "kana-only" to "katakana-only" in effective_script's docstring (さくらエミ is kana-only and licensed); added a forward-pointer from single_script to effective_script; moved _JA_SCRIPTS/_JA_PATTERN up beside _SCRIPT_PATTERNS and changed _JA_SCRIPTS from a sorted frozenset to a plain table-ordered tuple (nothing uses membership); made HIRAGANA's Policy docstring self-contained; folded the temporal-named Script-enum test into test_script_values_are_the_public_names; and trimmed the Han/Hangul-duplicate assertions out of the kana single_script test (renamed test_kana_singles_classify). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Native-script Chinese and Korean names now parse correctly by default, closing #271 for the 2.1.0 minor:
Policy.script_orders(default on): wholly-Han/Hangul names read family-first. Safe without language detection — Chinese, Japanese, and Korean all write family-first in native script; the languages differ, the convention doesn't.parse("김민준")→ family 김, given 민준. The 2015-census surname list ships as default vocabulary (nameparser/config/surnames.py) because hangul is unambiguously Korean and surnames are a closed set.locales.ZH(opt-in): Han segmentation for unspaced Chinese (毛泽东→ 毛 + 泽东, compounds like 夏侯 win over 夏). Opt-in because a zh surname list corrupts Japanese kanji names — 高橋一郎 must not split 高+橋一郎. Japanese is Japanese name segmentation via optional dependency (nameparser[ja]) #272's pluggable segmenter.AmbiguityKind.SEGMENTATION: recorded only when the vocabulary supported more than one split (남궁민수: 남궁+민수 vs 남+궁민수).HumanNamefacade as release-log-classified fixes.Design record: the 2026-07-27 script-scoped-order amendment (untracked specs dir). The rule it draws: script-conditional behavior only where the script itself — not statistics about it — determines the convention. Latin-script input is never affected ("Kim Min-jun" stays positional).
Review focal points
docs/migrate.rstdocuments all three changed shapes, including the quietest one (unspaced毛泽东movesfirst→lastwith a byte-identical string).Constantsis frozen; the escape hatch isPolicy(script_orders={}, segment_scripts=())(both fields must be cleared —customize.rstcalls out the trap).segmentso vocabulary can never change comma structure).str()re-render for hangul —str(HumanName("김민준"))is now"민준 김"; documented in migrate.rst.Policy/Lexiconpickles raise in 2.1.0 (guarded field-layout check); under Breaking Changes.Known residual risks
fix(#271)case-table rows are the real behavioral gate; theexpected_changes.tomlrule is pinned against_SCRIPT_RANGESbytest_regex_sync.pybut classifies nothing in the real run.locales.ZHmis-splits kanji names — the pack docstring andlocales.rstwarn; the recorded tradeoff is pinned as case rowzh_japanese_kanji_tradeoff.Test Plan
parse("김민준"),parse("毛 泽东"),parser_for(locales.ZH).parse("毛泽东"),HumanName("김민준")🤖 Generated with Claude Code