Word wrap get wrong line length while U+2018, U+2019, U+201C, U+201D are present

[qaqz111]

Issue Type: Bug

Word wrap get wrong line length calculated while these characters are present in line:
U+2018[‘](Left Single Quotation Mark)
U+2019[’](Right Single Quotation Mark)
U+201C[“](Left Double Quotation Mark)
U+201D[”](Right Double Quotation Mark)

These characters are wide characters and should take 2 alphabet characters' width, also known as fullwidth characters, they are different from U+0027' and U+0022["](Quotation Mark) which are narrow characters(halfwidth characters), see references below.

It looks like VSCode calculated these characters as narrow, thus while a long line with 2 or more these characters before wrap position, there will be 1 or more characters stay on the original line so that they are overlapped and covered by scroll bar.

Steps to reproduce this issue:

1. Press Ctrl+N to create a new file
2. Press Alt+Z to enable word wrap
3. Input some text like ABCDEFGHIJKLMNOPQRSTUVWXYZ
4. Paste the character above(anyone of them) multiple times at line beginning until line end reaches scroll bar
5. Keep pasting and you can see that the characters at line end was covered by scroll bar
6. Press Ctrl+F and toggle replace mode, replace the pasted character with 2 ASCII characters, you can see that the word wrap is working correctly now and no character is overlapped by scroll bar

References: Unicode® Standard Annex #11 - EAST ASIAN WIDTH
http://www.unicode.org/reports/tr11/
https://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt

And, my system is Windows10 x64 Simplified Chinese, I don't know if this is a bug under other systems of other language.

VS Code version: Code 1.32.3 (a3db5be, 2019-03-14T23:43:35.476Z)
OS version: Windows_NT x64 10.0.17763

[vscodebot]

(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

• Some punctuation causes incorrect wrapping when followed by quote. (#33236)
• Suggestion: render whitespace: show new line characters (#25525)

[qaqz111]

No, this issue is related to wrong character width calculating which is different from #33236, 2 issues may be occurred at same time.

[msftrncs]

I don't think this has anything to do with calculating width, since all the characters in a fixed width font are all the same width. Instead it seems that the first iteration of each said character are not being counted, possibly because they are also accepted for breaking characters.

Note the following weirdness occurring: (U+201C in this example)

Note line # 337 repeats twice.

[qaqz111]

since all the characters in a fixed width font are all the same width.

I'm sorry but this is not true. You can get the details following these reference links:
http://www.unicode.org/reports/tr11/
https://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt

That the characters are all the same width in your VSCode is possibly because the font you are using sets the characters mentioned above to half-width, but in fact they should be full-width. And there are many characters that are full-width, for example most of the CJK characters like 好好学习天天向上あいうえお, full-width digits and alphabets like １２３４ＡＢＣＤ, and punctuations used in these contexts like ，。：；‘’“”【】（）《》『』, etc.

[alexdima]

The width of the characters ‘’“” depends on the used font. They are not the same (semantically) as １２３４ＡＢＣＤ, which are specified in the Unicode standard as being fullwidth: e.g.

１ - Unicode Character 'FULLWIDTH DIGIT ONE' (U+FF11)
‘ - Unicode Character 'LEFT SINGLE QUOTATION MARK' (U+2018)

Our current wrapping implementation works OK for fullwidth characters because we measure one as a reference and use the width measured on that one for all fullwidth characters. But we cannot change ‘ to be treated as a fullwidth character, because it is not a fullwidth character across all fonts.

Let's therefore track in #2654

[qaqz111]

@alexdima Yes you are right about the width of characters like U+2018, and that's the point. According to the Unicode® Standard Annex #11 these characters are East Asian Ambiguous, therefore require additional information not contained in the character code to further resolve their width.

But VSCode seems to treat them as simple Narrow characters and give them halfwidth characters' width regardless the font used and therefore the horizontal scrollbar appears in word wrap mode.

To reproduce this issue:

1. Use a chinese or japanese font as first value of Editor: Font Family in preference which give these Ambiguous character wide width(e.g. Simsun in a Simplified Chinese Windows)
2. Input some text in VSCode editor like ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890
3. Toggle Word Wrap in VSCode through command or Alt+Z
4. Copy one or all of these characters ‘’“” and paste to line begin repeatly, and you'll see the horizontal scrollbar appears
5. Press Alt+Z repeatly and pay attention to the horizontal scrollbar you can see the issue

The point is, these characters ‘’“” in most East Asia context (e.g. chinese) they are always wide width as fullwidth characters, but VSCode can not process them correctly.