encoding: rudimentary TextDecoder support w/o ICU

Also split up the tests.
nodejs · Aug 2, 2017 · e07d268 · e07d268
1 parent 939818f
commit e07d268
Show file tree

Hide file tree

Showing 8 changed files with 428 additions and 268 deletions.
diff --git a/doc/api/errors.md b/doc/api/errors.md
@@ -826,6 +826,12 @@ would be possible by calling a callback more then once.
 Used when an attempt is made to use crypto features while Node.js is not
 compiled with OpenSSL crypto support.
 
+<a id="ERR_NO_ICU"></a>
+### ERR_NO_ICU
+
+Used when an attempt is made to use features that require [ICU][], while
+Node.js is not compiled with ICU support.
+
 <a id="ERR_NO_LONGER_SUPPORTED"></a>
 ### ERR_NO_LONGER_SUPPORTED
 
@@ -955,6 +961,7 @@ installed.
 [domains]: domain.html
 [event emitter-based]: events.html#events_class_eventemitter
 [file descriptors]: https://en.wikipedia.org/wiki/File_descriptor
+[ICU]: intl.html#intl_internationalization_support
 [online]: http://man7.org/linux/man-pages/man3/errno.3.html
 [stream-based]: stream.html
 [syscall]: http://man7.org/linux/man-pages/man2/syscall.2.html

diff --git a/doc/api/intl.md b/doc/api/intl.md
@@ -52,7 +52,7 @@ option:
 | [WHATWG URL Parser][]                   | partial (no IDN support)          | full                         | full                   | full
 | [`require('buffer').transcode()`][]     | none (function does not exist)    | full                         | full                   | full
 | [REPL][]                                | partial (inaccurate line editing) | full                         | full                   | full
-| [`require('util').TextDecoder`][]       | none (class does not exist)       | partial/full (depends on OS) | partial (Unicode-only) | full
+| [`require('util').TextDecoder`][]       | partial (basic encodings support) | partial/full (depends on OS) | partial (Unicode-only) | full
 
 *Note*: The "(not locale-aware)" designation denotes that the function carries
 out its operation just like the non-`Locale` version of the function, if one

diff --git a/doc/api/util.md b/doc/api/util.md
@@ -536,7 +536,7 @@ added: v8.0.0
 A Symbol that can be used to declare custom promisified variants of functions,
 see [Custom promisified functions][].
 
-### Class: util.TextDecoder
+## Class: util.TextDecoder
 <!-- YAML
 added: REPLACEME
 -->
@@ -555,23 +555,33 @@ while (buffer = getNextChunkSomehow()) {
 string += decoder.decode(); // end-of-stream
 ```
 
-#### WHATWG Supported Encodings
+### WHATWG Supported Encodings
 
 Per the [WHATWG Encoding Standard][], the encodings supported by the
 `TextDecoder` API are outlined in the tables below. For each encoding,
-one or more aliases may be used. Support for some encodings is enabled
-only when Node.js is using the full ICU data (see [Internationalization][]).
-`util.TextDecoder` is `undefined` when ICU is not enabled during build.
+one or more aliases may be used.
 
-##### Encodings Supported By Default
+Different Node.js build configurations support different sets of encodings.
+While a very basic set of encodings is supported even on Node.js builds without
+ICU enabled, support for some encodings is provided only when Node.js is built
+with ICU and using the full ICU data (see [Internationalization][]).
+
+#### Encodings Supported Without ICU
 
 | Encoding    | Aliases                           |
 | ----------- | --------------------------------- |
-| `'utf8'`    | `'unicode-1-1-utf-8'`, `'utf-8'`  |
-| `'utf-16be'`|                                   |
+| `'utf-8'`   | `'unicode-1-1-utf-8'`, `'utf8'`   |
 | `'utf-16le'`| `'utf-16'`                        |
 
-##### Encodings Requiring Full-ICU
+#### Encodings Supported by Default (With ICU)
+
+| Encoding    | Aliases                           |
+| ----------- | --------------------------------- |
+| `'utf-8'`   | `'unicode-1-1-utf-8'`, `'utf8'`   |
+| `'utf-16le'`| `'utf-16'`                        |
+| `'utf-16be'`|                                   |
+
+#### Encodings Requiring Full ICU Data
 
 | Encoding          | Aliases                          |
 | ----------------- | -------------------------------- |
@@ -613,13 +623,14 @@ only when Node.js is using the full ICU data (see [Internationalization][]).
 *Note*: The `'iso-8859-16'` encoding listed in the [WHATWG Encoding Standard][]
 is not supported.
 
-#### new TextDecoder([encoding[, options]])
+### new TextDecoder([encoding[, options]])
 
 * `encoding` {string} Identifies the `encoding` that this `TextDecoder` instance
   supports. Defaults to `'utf-8'`.
 * `options` {Object}
   * `fatal` {boolean} `true` if decoding failures are fatal. Defaults to
-    `false`.
+    `false`. This option is only supported when ICU is enabled (see
+    [Internationalization][]).
   * `ignoreBOM` {boolean} When `true`, the `TextDecoder` will include the byte
      order mark in the decoded result. When `false`, the byte order mark will
      be removed from the output. This option is only used when `encoding` is
@@ -628,7 +639,7 @@ is not supported.
 Creates an new `TextDecoder` instance. The `encoding` may specify one of the
 supported encodings or an alias.
 
-#### textDecoder.decode([input[, options]])
+### textDecoder.decode([input[, options]])
 
 * `input` {ArrayBuffer|DataView|TypedArray} An `ArrayBuffer`, `DataView` or
   Typed Array instance containing the encoded data.
@@ -644,49 +655,55 @@ internally and emitted after the next call to `textDecoder.decode()`.
 If `textDecoder.fatal` is `true`, decoding errors that occur will result in a
 `TypeError` being thrown.
 
-#### textDecoder.encoding
+### textDecoder.encoding
 
-* Value: {string}
+* {string}
 
 The encoding supported by the `TextDecoder` instance.
 
-#### textDecoder.fatal
+### textDecoder.fatal
 
-* Value: {boolean}
+* {boolean}
 
 The value will be `true` if decoding errors result in a `TypeError` being
 thrown.
 
-#### textDecoder.ignoreBOM
+### textDecoder.ignoreBOM
 
-* Value: {boolean}
+* {boolean}
 
 The value will be `true` if the decoding result will include the byte order
 mark.
 
-### Class: util.TextEncoder
+## Class: util.TextEncoder
 <!-- YAML
 added: REPLACEME
 -->
 
 > Stability: 1 - Experimental
 
 An implementation of the [WHATWG Encoding Standard][] `TextEncoder` API. All
-instances of `TextEncoder` only support `UTF-8` encoding.
+instances of `TextEncoder` only support UTF-8 encoding.
 
 ```js
 const encoder = new TextEncoder();
 const uint8array = encoder.encode('this is some data');
 ```
 
-#### textEncoder.encode([input])
+### textEncoder.encode([input])
 
 * `input` {string} The text to encode. Defaults to an empty string.
 * Returns: {Uint8Array}
 
-UTF-8 Encodes the `input` string and returns a `Uint8Array` containing the
+UTF-8 encodes the `input` string and returns a `Uint8Array` containing the
 encoded bytes.
 
+### textDecoder.encoding
+
+* {string}
+
+The encoding supported by the `TextEncoder` instance. Always set to `'utf-8'`.
+
 ## Deprecated APIs
 
 The following APIs have been deprecated and should no longer be used. Existing