You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Match namereplace private name records to CPython UCD tables - #9827
Port CPython 3.15's UCD-file-order private name records used by namereplace.
CPython's Tools/unicode/makeunicodedata.py places name aliases at U+F0000 and named sequences at U+F0200. Its private getname callback includes these records for namereplace, while public unicodedata.name excludes them. Generate equivalent ordered tables from the already vendored UCD files, keeping existing public lookups unchanged.
For example U+F0000 now produces \N{NULL} and U+F0200 produces \N{KEYCAP NUMBER SIGN}, rather than numeric escapes.
Validation
Compared every record against CPython 3.15.2: 481 aliases and 461 named sequences, including public name(..., None) and out-of-range boundaries.
cargo test -p rustpython-unicode --all-features: 5 passed.
Full VM runtime tests not yet run on this patch due limited disk; draft pending CI.
Scope
This does not fix the two remaining test_unicodedata expected failures concerning lazy module import/reload and compiler name escapes. Those markers remain unchanged. Proper shared provider lifecycle and Ruff parser callback need separate work, without accepting unsafe foreign pointers or introducing a fake capsule ABI. The casing/checksum fix is separate in #9824; no casing logic is duplicated here.
Summary by CodeRabbit
New Features
Unicode name replacement now recognizes designated code points for aliases and named sequences, producing their corresponding names in encoded output.
Other valid Unicode characters continue to use their standard character names, while invalid code points retain the existing numeric escape behavior.
Tests
Added coverage for alias and named-sequence replacements, range boundaries, and fallback behavior.
No actionable comments were generated in the recent review.
Walkthrough
Name replacement now resolves designated private-use code points through ordered alias and named-sequence tables. Other valid Unicode scalar values use their Unicode character names, and numeric escape fallbacks remain unchanged.
Changes: Unicode name replacement
Layer / File(s)
Summary
Generate and resolve replacement names crates/unicode/build.rs, crates/unicode/src/data.rs
The generator emits alias and named-sequence names in input order and asserts that the alias count is below 0x200. name_for_namereplace resolves the corresponding private-use ranges and falls back to Unicode character names for other valid scalar values. Tests cover table ordering, range boundaries, and fallback behavior.
Use replacement-name lookup in encoding crates/common/src/encodings.rs, extra_tests/snippets/stdlib_unicode_shared.py
NameReplace::handle_encode_error uses name_for_namereplace. Tests check the replacement output for U+F0000 and U+F0200 and confirm that unicodedata.name returns None for both.
This change makes namereplace produce the CPython-compatible names for the two designated private-use code points. Public Unicode name lookups are unchanged, and no actionable merge-blocking risk remains beyond CI confirmation.
Pre-merge checks: 4
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title names the main change: matching namereplace private-use name records to the CPython UCD table order. The raw summary shows the encodings.rs lookup switch, the build.rs ordered tables, the ne…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the main change: matching namereplace private-use name records to the CPython UCD table order. The raw summary shows the encodings.rs lookup switch, the build.rs ordered tables, the new name_for_namereplace function, and the added tests all serve that goal. The title is specific and does not list files or use vague terms. It is 60 characters, within a reasonable length.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Reviewing files that changed from the base of the PR and between cc6fb57 and 255aa03.
Files selected for processing: 4
crates/common/src/encodings.rs
crates/unicode/build.rs
crates/unicode/src/data.rs
extra_tests/snippets/stdlib_unicode_shared.py
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.
I think it may help fix some of the EAW discrepancies that I couldn't figure out as well.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port CPython 3.15's UCD-file-order private name records used by
namereplace.CPython's
Tools/unicode/makeunicodedata.pyplaces name aliases at U+F0000 and named sequences at U+F0200. Its private getname callback includes these records fornamereplace, while publicunicodedata.nameexcludes them. Generate equivalent ordered tables from the already vendored UCD files, keeping existing public lookups unchanged.For example U+F0000 now produces
\N{NULL}and U+F0200 produces\N{KEYCAP NUMBER SIGN}, rather than numeric escapes.Validation
name(..., None)and out-of-range boundaries.cargo test -p rustpython-unicode --all-features: 5 passed.cargo check -p rustpython-commonpassed.Scope
This does not fix the two remaining
test_unicodedataexpected failures concerning lazy module import/reload and compiler name escapes. Those markers remain unchanged. Proper shared provider lifecycle and Ruff parser callback need separate work, without accepting unsafe foreign pointers or introducing a fake capsule ABI. The casing/checksum fix is separate in #9824; no casing logic is duplicated here.Summary by CodeRabbit