Format hex code in unicode escape sequences in string literals #2916

Shivansh-007 · 2022-03-13T10:55:05Z

Closes #2067
Closes #2828

Checklist - did you ...

Add a CHANGELOG entry if necessary?
Add / update tests if necessary?
Add new / update outdated documentation? -> n/a

According to the table at https://docs.python.org/3/reference/lexical_analysis.html\#string-and-bytes-literals

felix-hilden

Thanks for the PR once again! A comment and some nits below 👍 Let's discuss.

src/black/linegen.py

src/black/mode.py

felix-hilden · 2022-03-14T16:44:47Z

src/black/strings.py

+
+
+def normalize_unicode_escape_sequences(leaf: Leaf) -> None:
+    """Replace hex codes in Unicode escape sequences with lowercase representation."""


This will have to be thought out still, as this comment points out. My two cents: I prefer upper case, and since Black formats hex numbers to upper already I think it would be consistent. The Python repr argument is solid too, but we should think about changing hex literals as well then.

I'd rather not change hex numbers, we already changed our mind there a few times.

So if we're not changing numbers (which I agree with), do y'all share the concern for consistency?

My comments read a bit ambiguously. So to be clear, I'm proposing that we switch the formatting to be upper case to be consistent with hex numbers. Y'all in?

src/black/strings.py

JelleZijlstra · 2022-03-16T03:24:10Z

src/black/strings.py

+
+
+def normalize_unicode_escape_sequences(leaf: Leaf) -> None:
+    """Replace hex codes in Unicode escape sequences with lowercase representation."""


I'd rather not change hex numbers, we already changed our mind there a few times.

test.py

…erals

github-actions · 2022-03-16T04:11:14Z

diff-shades results comparing this PR (1511959) to main (4e3303f). The full diff is available in the logs under the "Generate HTML diff report" step.

╭──────────────────────── Summary ────────────────────────╮
│ 5 projects & 38 files changed / 290 changes [+145/-145] │
│                                                         │
│ ... out of 2 363 850 lines, 11 046 files & 23 projects  │
╰─────────────────────────────────────────────────────────╯

Differences found.

What is this? | Workflow run | diff-shades documentation

src/black/strings.py

CHANGES.md

Co-authored-by: Jelle Zijlstra <[email protected]>

ichard26

I won't comment on the actual formatting style, but I got quite a few other suggestions. Not sure if this is too minor, but I'd recommend checking this is covered in the Black code style documentation!

Thanks again!

src/black/strings.py

ichard26

I forgot to mark my review as "request changes" which is relevant since this PR can still crash.

ichard26 · 2022-06-27T20:22:14Z

Hi @Shivansh-007, are you still able to and interested in working on this PR? If not, just lemme know and I'd be happy to pick it up!

Also, use named groups

JelleZijlstra · 2022-12-18T16:33:06Z

I brought this PR up to date, applied @ichard26's review suggestions, and fixed a few more things I noticed. I think this PR is now good to go unless we change our mind to go with uppercase (#2067).

JelleZijlstra · 2022-12-18T16:42:01Z

I determined the legal characters in \N escapes by doing something like [unicodedata.name(chr(i)) for i in range(65536)] (but ignoring invalid characters) and taking the set of all characters in the output. The length of the names ranged from 3 to 83. However, \N also accepts aliases and I'm not sure how to get a list of all of those; the Python docs point to https://www.unicode.org/Public/14.0.0/ucd/NameAliases.txt but that doesn't include the "ox" alias for 🐂. I manually verified that there are no one-character aliases.

Jackenmen · 2022-12-18T16:50:06Z

However, \N also accepts aliases and I'm not sure how to get a list of all of those; the Python docs point to unicode.org/Public/14.0.0/ucd/NameAliases.txt but that doesn't include the "ox" alias for 🐂

"ox" is the base name for 🐂 so it's returned by unicodedata.name().

JelleZijlstra · 2022-12-18T16:52:15Z

Ah thanks, I should have gone past 65536 to include astral characters. That increases the length range from 2 to 88 but doesn't add more characters to the set of characters that appear in names.

JelleZijlstra · 2022-12-18T16:53:17Z

Also the longest names are

In [13]: [n for n in names if len(n) > 80]
Out[13]: 
['ARABIC LIGATURE UIGHUR KIRGHIZ YEH WITH HAMZA ABOVE WITH ALEF MAKSURA ISOLATED FORM',
 'ARABIC LIGATURE UIGHUR KIRGHIZ YEH WITH HAMZA ABOVE WITH ALEF MAKSURA INITIAL FORM',
 'BOX DRAWINGS LIGHT DIAGONAL UPPER CENTRE TO MIDDLE LEFT AND MIDDLE RIGHT TO LOWER CENTRE',
 'BOX DRAWINGS LIGHT DIAGONAL UPPER CENTRE TO MIDDLE RIGHT AND MIDDLE LEFT TO LOWER CENTRE',
 'BOX DRAWINGS LIGHT DIAGONAL UPPER CENTRE TO MIDDLE RIGHT TO LOWER CENTRE TO MIDDLE LEFT',
 'BOX DRAWINGS LIGHT DIAGONAL UPPER CENTRE TO MIDDLE LEFT TO LOWER CENTRE TO MIDDLE RIGHT',
 'BOX DRAWINGS LIGHT DIAGONAL MIDDLE LEFT TO UPPER CENTRE TO MIDDLE RIGHT TO LOWER CENTRE',
 'BOX DRAWINGS LIGHT DIAGONAL MIDDLE RIGHT TO UPPER CENTRE TO MIDDLE LEFT TO LOWER CENTRE']

stale

Noticeable style changes: 1. Parenthesize multiple context managers psf#3489. The following style changes are temporarily disabled when `--preview` is used together with `--pyink`: 2. Format unicode escape sequences psf#2916. 3. Parenthesize conditional expressions psf#2278. PiperOrigin-RevId: 507485670

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [black](https://github.com/psf/black) ([changelog](https://github.com/psf/black/blob/main/CHANGES.md)) | dependency-groups | major | `<25.0.0,>=24.10.0` -> `<25.2.0,>=25.1.0` | --- ### Release Notes <details> <summary>psf/black (black)</summary> ### [`v25.1.0`](https://github.com/psf/black/blob/HEAD/CHANGES.md#2510) [Compare Source](psf/black@24.10.0...25.1.0) ##### Highlights This release introduces the new 2025 stable style ([#4558](psf/black#4558)), stabilizing the following changes: - Normalize casing of Unicode escape characters in strings to lowercase ([#2916](psf/black#2916)) - Fix inconsistencies in whether certain strings are detected as docstrings ([#4095](psf/black#4095)) - Consistently add trailing commas to typed function parameters ([#4164](psf/black#4164)) - Remove redundant parentheses in if guards for case blocks ([#4214](psf/black#4214)) - Add parentheses to if clauses in case blocks when the line is too long ([#4269](psf/black#4269)) - Whitespace before `# fmt: skip` comments is no longer normalized ([#4146](psf/black#4146)) - Fix line length computation for certain expressions that involve the power operator ([#4154](psf/black#4154)) - Check if there is a newline before the terminating quotes of a docstring ([#4185](psf/black#4185)) - Fix type annotation spacing between `*` and more complex type variable tuple ([#4440](psf/black#4440)) The following changes were not in any previous release: - Remove parentheses around sole list items ([#4312](psf/black#4312)) - Generic function definitions are now formatted more elegantly: parameters are split over multiple lines first instead of type parameter definitions ([#4553](psf/black#4553)) ##### Stable style - Fix formatting cells in IPython notebooks with magic methods and starting or trailing empty lines ([#4484](psf/black#4484)) - Fix crash when formatting `with` statements containing tuple generators/unpacking ([#4538](psf/black#4538)) ##### Preview style - Fix/remove string merging changing f-string quotes on f-strings with internal quotes ([#4498](psf/black#4498)) - Collapse multiple empty lines after an import into one ([#4489](psf/black#4489)) - Prevent `string_processing` and `wrap_long_dict_values_in_parens` from removing parentheses around long dictionary values ([#4377](psf/black#4377)) - Move `wrap_long_dict_values_in_parens` from the unstable to preview style ([#4561](psf/black#4561)) ##### Packaging - Store license identifier inside the `License-Expression` metadata field, see [PEP 639](https://peps.python.org/pep-0639/). ([#4479](psf/black#4479)) ##### Performance - Speed up the `is_fstring_start` function in Black's tokenizer ([#4541](psf/black#4541)) ##### Integrations - If using stdin with `--stdin-filename` set to a force excluded path, stdin won't be formatted. ([#4539](psf/black#4539)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.tainton.uk/repos/PwnedPW/pulls/283 Reviewed-by: Luke Tainton <[email protected]> Co-authored-by: Renovate [BOT] <[email protected]> Co-committed-by: Renovate [BOT] <[email protected]>

Shivansh-007 added 6 commits March 13, 2022 12:49

Format hex code in unicode escape sequences in string literals

add30b8

Format \N character name escapes with uppercased literals

483fc15

Fix formatting with correct length for each format

cc48d2d

According to the table at https://docs.python.org/3/reference/lexical_analysis.html\#string-and-bytes-literals

Add changelog

f1dbc96

Move feature to preview styling only

ef442a6

Fix typo

2ada012

felix-hilden reviewed Mar 14, 2022

View reviewed changes

JelleZijlstra mentioned this pull request Mar 15, 2022

Remove unnecessary parentheses from with statements #2926

Merged

3 tasks

JelleZijlstra reviewed Mar 16, 2022

View reviewed changes

src/black/strings.py Outdated Show resolved Hide resolved

JelleZijlstra reviewed Mar 16, 2022

View reviewed changes

Shivansh-007 added 4 commits March 16, 2022 09:06

Change Match[AnyStr] to Match[str]

125ebec

Make UNICODE_RE Final and accept multiline strings

af86102

Reword regex comments to use 'character'

69c9664

Merge remote-tracking branch 'upstream/main' into format/hex-code-lit…

7d0e548

…erals

Shivansh-007 force-pushed the format/hex-code-literals branch from c7fc77c to 7d0e548 Compare March 16, 2022 04:00

Shivansh-007 requested review from JelleZijlstra and felix-hilden March 16, 2022 04:00

JelleZijlstra reviewed Mar 16, 2022

View reviewed changes

src/black/strings.py Outdated Show resolved Hide resolved

Shivansh-007 and others added 2 commits March 16, 2022 10:04

ITS RE.VERBOSE NOT RE.MULTILINE?!

52bd904

Merge branch 'main' into format/hex-code-literals

a5c4e62

JelleZijlstra reviewed Mar 24, 2022

View reviewed changes

CHANGES.md Show resolved Hide resolved

JelleZijlstra approved these changes Mar 24, 2022

View reviewed changes

JelleZijlstra self-assigned this Mar 24, 2022

Update CHANGES.md

221995e

Co-authored-by: Jelle Zijlstra <[email protected]>

ichard26 reviewed Mar 24, 2022

View reviewed changes

src/black/strings.py Show resolved Hide resolved

src/black/strings.py Outdated Show resolved Hide resolved

src/black/strings.py Outdated Show resolved Hide resolved

src/black/strings.py Outdated Show resolved Hide resolved

ichard26 previously requested changes Mar 25, 2022

View reviewed changes

Merge branch 'main' into format/hex-code-literals

d4dde2e

JelleZijlstra removed their assignment Apr 2, 2022

JelleZijlstra added 7 commits December 18, 2022 07:54

Merge branch 'main' into format/hex-code-literals

3557faf

CR improvements

77a48e6

Also, use named groups

fix lint

1b9d5fd

fix my sloppy code

3c24427

fix the new test; \U requires exactly 8 digits

9f35b61

fix \N escapes

27d2d86

add a test

420a8f9

JelleZijlstra requested a review from ichard26 December 18, 2022 16:27

JelleZijlstra removed help wanted Extra attention is needed S: up for grabs (PR only) Available for anyone to work on as the PR author is busy or unreachable. labels Dec 18, 2022

JelleZijlstra self-assigned this Dec 18, 2022

bytes tests

296cdb9

JelleZijlstra added 2 commits December 29, 2022 15:13

Merge branch 'main' into format/hex-code-literals

625c085

Merge branch 'main' into format/hex-code-literals

1511959

JelleZijlstra merged commit eabff67 into psf:main Jan 22, 2023

JelleZijlstra mentioned this pull request Feb 13, 2023

Feedback on lowercase \U, \u, \x escapes introduced in 23.1 preview style #2916 #3568

Closed

hauntsaninja mentioned this pull request Nov 13, 2023

Setting the 2024 stable style #4042

Closed

konstin mentioned this pull request Nov 14, 2023

🏖️ Black 2024 Preview Style astral-sh/ruff#8678

Closed

28 tasks

MichaReiser mentioned this pull request Dec 26, 2023

Normalise Hex and unicode escape sequences in string astral-sh/ruff#9280

Merged

JelleZijlstra mentioned this pull request Dec 4, 2024

Setting the 2025 stable style #4522

Closed

notatallshaw mentioned this pull request Jan 30, 2025

Update Black (2025 style) pypa/pip#13193

Merged

macbre mentioned this pull request Feb 6, 2025

Bump black from 24.8.0 to 25.1.0 macbre/mediawiki-dump#267

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Format hex code in unicode escape sequences in string literals #2916

Format hex code in unicode escape sequences in string literals #2916

Shivansh-007 commented Mar 13, 2022

felix-hilden left a comment

felix-hilden Mar 14, 2022

JelleZijlstra Mar 16, 2022

felix-hilden Mar 24, 2022

felix-hilden Jun 30, 2022

JelleZijlstra Mar 16, 2022

github-actions bot commented Mar 16, 2022 •

edited

Loading

ichard26 left a comment

ichard26 left a comment

ichard26 commented Jun 27, 2022

JelleZijlstra commented Dec 18, 2022

JelleZijlstra commented Dec 18, 2022

Jackenmen commented Dec 18, 2022

JelleZijlstra commented Dec 18, 2022

JelleZijlstra commented Dec 18, 2022



		def normalize_unicode_escape_sequences(leaf: Leaf) -> None:
		"""Replace hex codes in Unicode escape sequences with lowercase representation."""

Format hex code in unicode escape sequences in string literals #2916

Format hex code in unicode escape sequences in string literals #2916

Conversation

Shivansh-007 commented Mar 13, 2022

Checklist - did you ...

felix-hilden left a comment

Choose a reason for hiding this comment

felix-hilden Mar 14, 2022

Choose a reason for hiding this comment

JelleZijlstra Mar 16, 2022

Choose a reason for hiding this comment

felix-hilden Mar 24, 2022

Choose a reason for hiding this comment

felix-hilden Jun 30, 2022

Choose a reason for hiding this comment

JelleZijlstra Mar 16, 2022

Choose a reason for hiding this comment

github-actions bot commented Mar 16, 2022 • edited Loading

ichard26 left a comment

Choose a reason for hiding this comment

ichard26 left a comment

Choose a reason for hiding this comment

ichard26 commented Jun 27, 2022

JelleZijlstra commented Dec 18, 2022

JelleZijlstra commented Dec 18, 2022

Jackenmen commented Dec 18, 2022

JelleZijlstra commented Dec 18, 2022

JelleZijlstra commented Dec 18, 2022

github-actions bot commented Mar 16, 2022 •

edited

Loading