From 474dfc457f5592906fcfc5277cbcff7190fef2b3 Mon Sep 17 00:00:00 2001
From: Anish Athalye <me@anishathalye.com>
Date: Sun, 8 Dec 2019 11:18:04 -0500
Subject: [PATCH] Fix reader for Unicode code points over 0xFFFF (#351)

This patch fixes the handling of inputs with Unicode code points over
0xFFFF when running on a Python 2 that does not have UCS-4 support
(which certain distributions still ship, e.g. macOS).

When Python is compiled without UCS-4 support, it uses UCS-2. In this
situation, non-BMP Unicode characters, which have code points over
0xFFFF, are represented as surrogate pairs. For example, if we take
u'\U0001f3d4', it will be represented as the surrogate pair
u'\ud83c\udfd4'. This can be seen by running, for example:

    [i for i in u'\U0001f3d4']

In PyYAML, the reader uses a function `check_printable` to validate
inputs, making sure that they only contain printable characters. Prior
to this patch, on UCS-2 builds, it incorrectly identified surrogate
pairs as non-printable.

It would be fairly natural to write a regular expression that captures
strings that contain only *printable* characters, as opposed to
*non-printable* characters (as identified by the old code, so not
excluding surrogate pairs):

    PRINTABLE = re.compile(u'^[\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]*$')

Adding support for surrogate pairs to this would be straightforward,
adding the option of having a surrogate high followed by a surrogate low
(`[\uD800-\uDBFF][\uDC00-\uDFFF]`):

    PRINTABLE = re.compile(u'^(?:[\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]|[\uD800-\uDBFF][\uDC00-\uDFFF])*$')

Then, this regex could be used as follows:

    def check_printable(self, data):
        if not self.PRINTABLE.match(data):
            raise ReaderError(...)

However, matching printable strings, rather than searching for
non-printable characters as the code currently does, would have the
disadvantage of not identifying the culprit character (we wouldn't get
the position and the actual non-printable character from a lack of a
regex match).

Instead, we can modify the NON_PRINTABLE regex to allow legal surrogate
pairs. We do this by removing surrogate pairs from the existing
character set and adding the following options for illegal uses of
surrogate code points:

- Surrogate low that doesn't follow a surrogate high (either a surrogate
  low at the start of a string, or a surrogate low that follows a
  character that's not a surrogate high):

    (?:^|[^\uD800-\uDBFF])[\uDC00-\uDFFF]

- Surrogate high that isn't followed by a surrogate low (either a
  surrogate high at the end of a string, or a surrogate high that is
  followed by a character that's not a surrogate low):

    [\uD800-\uDBFF](?:[^\uDC00-\uDFFF]|$)

The behavior of this modified regex should match the one that is used
when Python is built with UCS-4 support.
---
 lib/yaml/reader.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/yaml/reader.py b/lib/yaml/reader.py
index b2f10b09..4b377d61 100644
--- a/lib/yaml/reader.py
+++ b/lib/yaml/reader.py
@@ -139,7 +139,7 @@ def determine_encoding(self):
     if has_ucs4:
         NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD\U00010000-\U0010ffff]')
     else:
-        NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]')
+        NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uFFFD]|(?:^|[^\uD800-\uDBFF])[\uDC00-\uDFFF]|[\uD800-\uDBFF](?:[^\uDC00-\uDFFF]|$)')
     def check_printable(self, data):
         match = self.NON_PRINTABLE.search(data)
         if match: