doc: add documentation for brotli support

Backport-PR-URL: #27681 PR-URL: #24938 Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Jan Krems <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Myles Borins <[email protected]> Reviewed-By: Ali Ijaz Sheikh <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>
nodejs · May 16, 2019 · 93edf90 · 93edf90
1 parent c670358
commit 93edf90
Show file tree

Hide file tree

Showing 3 changed files with 211 additions and 13 deletions.
diff --git a/doc/api/errors.md b/doc/api/errors.md
@@ -624,16 +624,16 @@ An attempt was made to register something that is not a function as an
 The type of an asynchronous resource was invalid. Note that users are also able
 to define their own types if using the public embedder API.
 
+<a id="ERR_BROTLI_COMPRESSION_FAILED"></a>
+### ERR_BROTLI_COMPRESSION_FAILED
+
+Data passed to a Brotli stream was not successfully compressed.
+
 <a id="ERR_BROTLI_INVALID_PARAM"></a>
 ### ERR_BROTLI_INVALID_PARAM
 
 An invalid parameter key was passed during construction of a Brotli stream.
 
-<a id="ERR_BROTLI_COMPRESSION_FAILED">
-### ERR_BROTLI_COMPRESSION_FAILED
-
-Data passed to a Brotli stream was not successfully compressed.
-
 <a id="ERR_BUFFER_OUT_OF_BOUNDS"></a>
 ### ERR_BUFFER_OUT_OF_BOUNDS
 

diff --git a/doc/api/zlib.md b/doc/api/zlib.md
@@ -5,7 +5,7 @@
 > Stability: 2 - Stable
 
 The `zlib` module provides compression functionality implemented using Gzip and
-Deflate/Inflate. It can be accessed using:
+Deflate/Inflate, as well as Brotli. It can be accessed using:
 
 ```js
 const zlib = require('zlib');
@@ -54,8 +54,8 @@ and/or unrecoverable and catastrophic memory fragmentation.
 
 ## Compressing HTTP requests and responses
 
-The `zlib` module can be used to implement support for the `gzip` and `deflate`
-content-encoding mechanisms defined by
+The `zlib` module can be used to implement support for the `gzip`, `deflate`
+and `br` content-encoding mechanisms defined by
 [HTTP](https://tools.ietf.org/html/rfc7230#section-4.2).
 
 The HTTP [`Accept-Encoding`][] header is used within an http request to identify
@@ -76,12 +76,15 @@ const fs = require('fs');
 const request = http.get({ host: 'example.com',
                            path: '/',
                            port: 80,
-                           headers: { 'Accept-Encoding': 'gzip,deflate' } });
+                           headers: { 'Accept-Encoding': 'br,gzip,deflate' } });
 request.on('response', (response) => {
   const output = fs.createWriteStream('example.com_index.html');
 
   switch (response.headers['content-encoding']) {
-    // or, just use zlib.createUnzip() to handle both cases
+    case 'br':
+      response.pipe(zlib.createBrotliDecompress()).pipe(output);
+      break;
+    // Or, just use zlib.createUnzip() to handle both of the following cases:
     case 'gzip':
       response.pipe(zlib.createGunzip()).pipe(output);
       break;
@@ -117,6 +120,9 @@ http.createServer((request, response) => {
   } else if (/\bgzip\b/.test(acceptEncoding)) {
     response.writeHead(200, { 'Content-Encoding': 'gzip' });
     raw.pipe(zlib.createGzip()).pipe(response);
+  } else if (/\bbr\b/.test(acceptEncoding)) {
+    response.writeHead(200, { 'Content-Encoding': 'br' });
+    raw.pipe(zlib.createBrotliCompress()).pipe(response);
   } else {
     response.writeHead(200, {});
     raw.pipe(response);
@@ -136,6 +142,7 @@ const buffer = Buffer.from('eJzT0yMA', 'base64');
 
 zlib.unzip(
   buffer,
+  // For Brotli, the equivalent is zlib.constants.BROTLI_OPERATION_FLUSH.
   { finishFlush: zlib.constants.Z_SYNC_FLUSH },
   (err, buffer) => {
     if (!err) {
@@ -156,6 +163,8 @@ decompressed result is valid.
 
 <!--type=misc-->
 
+### For zlib-based streams
+
 From `zlib/zconf.h`, modified to Node.js's usage:
 
 The memory requirements for deflate are (in bytes):
@@ -194,6 +203,16 @@ fewer calls to `zlib` because it will be able to process more data on
 each `write` operation. So, this is another factor that affects the
 speed, at the cost of memory usage.
 
+### For Brotli-based streams
+
+There are equivalents to the zlib options for Brotli-based streams, although
+these options have different ranges than the zlib ones:
+
+- zlib’s `level` option matches Brotli’s `BROTLI_PARAM_QUALITY` option.
+- zlib’s `windowBits` option matches Brotli’s `BROTLI_PARAM_LGWIN` option.
+
+See [below][Brotli parameters] for more details on Brotli-specific options.
+
 ## Flushing
 
 Calling [`.flush()`][] on a compression stream will make `zlib` return as much
@@ -231,6 +250,8 @@ added: v0.5.8
 
 <!--type=misc-->
 
+### zlib constants
+
 All of the constants defined in `zlib.h` are also defined on
 `require('zlib').constants`. In the normal course of operations, it will not be
 necessary to use these constants. They are documented so that their presence is
@@ -281,6 +302,77 @@ Compression strategy.
 * `zlib.constants.Z_FIXED`
 * `zlib.constants.Z_DEFAULT_STRATEGY`
 
+### Brotli constants
+<!-- YAML
+added: REPLACEME
+-->
+
+There are several options and other constants available for Brotli-based
+streams:
+
+#### Flush operations
+
+The following values are valid flush operations for Brotli-based streams:
+
+* `zlib.constants.BROTLI_OPERATION_PROCESS` (default for all operations)
+* `zlib.constants.BROTLI_OPERATION_FLUSH` (default when calling `.flush()`)
+* `zlib.constants.BROTLI_OPERATION_FINISH` (default for the last chunk)
+* `zlib.constants.BROTLI_OPERATION_EMIT_METADATA`
+  * This particular operation may be hard to use in a Node.js context,
+     as the streaming layer makes it hard to know which data will end up
+     in this frame. Also, there is currently no way to consume this data through
+     the Node.js API.
+
+#### Compressor options
+
+There are several options that can be set on Brotli encoders, affecting
+compression efficiency and speed. Both the keys and the values can be accessed
+as properties of the `zlib.constants` object.
+
+The most important options are:
+
+* `BROTLI_PARAM_MODE`
+  * `BROTLI_MODE_GENERIC` (default)
+  * `BROTLI_MODE_TEXT`, adjusted for UTF-8 text
+  * `BROTLI_MODE_FONT`, adjusted for WOFF 2.0 fonts
+* `BROTLI_PARAM_QUALITY`
+  * Ranges from `BROTLI_MIN_QUALITY` to `BROTLI_MAX_QUALITY`,
+    with a default of `BROTLI_DEFAULT_QUALITY`.
+* `BROTLI_PARAM_SIZE_HINT`
+  * Integer value representing the expected input size;
+    defaults to `0` for an unknown input size.
+
+The following flags can be set for advanced control over the compression
+algorithm and memory usage tuning:
+
+* `BROTLI_PARAM_LGWIN`
+  * Ranges from `BROTLI_MIN_WINDOW_BITS` to `BROTLI_MAX_WINDOW_BITS`,
+    with a default of `BROTLI_DEFAULT_WINDOW`, or up to
+    `BROTLI_LARGE_MAX_WINDOW_BITS` if the `BROTLI_PARAM_LARGE_WINDOW` flag
+    is set.
+* `BROTLI_PARAM_LGBLOCK`
+  * Ranges from `BROTLI_MIN_INPUT_BLOCK_BITS` to `BROTLI_MAX_INPUT_BLOCK_BITS`.
+* `BROTLI_PARAM_DISABLE_LITERAL_CONTEXT_MODELING`
+  * Boolean flag that decreases compression ratio in favour of
+    decompression speed.
+* `BROTLI_PARAM_LARGE_WINDOW`
+  * Boolean flag enabling “Large Window Brotli” mode (not compatible with the
+    Brotli format as standardized in [RFC 7932][]).
+* `BROTLI_PARAM_NPOSTFIX`
+  * Ranges from `0` to `BROTLI_MAX_NPOSTFIX`.
+* `BROTLI_PARAM_NDIRECT`
+  * Ranges from `0` to `15 << NPOSTFIX` in steps of `1 << NPOSTFIX`.
+
+#### Decompressor options
+
+These advanced options are available for controlling decompression:
+
+* `BROTLI_DECODER_PARAM_DISABLE_RING_BUFFER_REALLOCATION`
+  * Boolean flag that affects internal memory allocation patterns.
+* `BROTLI_DECODER_PARAM_LARGE_WINDOW`
+  * Boolean flag enabling “Large Window Brotli” mode (not compatible with the
+    Brotli format as standardized in [RFC 7932][]).
+
 ## Class: Options
 <!-- YAML
 added: v0.11.1
@@ -298,7 +390,7 @@ changes:
 
 <!--type=misc-->
 
-Each class takes an `options` object. All options are optional.
+Each zlib-based class takes an `options` object. All options are optional.
 
 Note that some options are only relevant when compressing, and are
 ignored by the decompression classes.
@@ -317,6 +409,47 @@ ignored by the decompression classes.
 See the description of `deflateInit2` and `inflateInit2` at
 <https://zlib.net/manual.html#Advanced> for more information on these.
 
+## Class: BrotliOptions
+<!-- YAML
+added: REPLACEME
+-->
+
+<!--type=misc-->
+
+Each Brotli-based class takes an `options` object. All options are optional.
+
+* `flush` {integer} **Default:** `zlib.constants.BROTLI_OPERATION_PROCESS`
+* `finishFlush` {integer} **Default:** `zlib.constants.BROTLI_OPERATION_FINISH`
+* `chunkSize` {integer} **Default:** `16 * 1024`
+* `params` {Object} Key-value object containing indexed [Brotli parameters][].
+
+For example:
+
+```js
+const stream = zlib.createBrotliCompress({
+  chunkSize: 32 * 1024,
+  params: {
+    [zlib.constants.BROTLI_PARAM_MODE]: zlib.constants.BROTLI_MODE_TEXT,
+    [zlib.constants.BROTLI_PARAM_QUALITY]: 4,
+    [zlib.constants.BROTLI_PARAM_SIZE_HINT]: fs.statSync(inputFile).size
+  }
+});
+```
+
+## Class: zlib.BrotliCompress
+<!-- YAML
+added: REPLACEME
+-->
+
+Compress data using the Brotli algorithm.
+
+## Class: zlib.BrotliDecompress
+<!-- YAML
+added: REPLACEME
+-->
+
+Decompress data using the Brotli algorithm.
+
 ## Class: zlib.Deflate
 <!-- YAML
 added: v0.5.8
@@ -389,9 +522,13 @@ added: v0.5.8
 Decompress either a Gzip- or Deflate-compressed stream by auto-detecting
 the header.
 
-## Class: zlib.Zlib
+## Class: zlib.ZlibBase
 <!-- YAML
 added: v0.5.8
+changes:
+  - version: REPLACEME
+    pr-url: https://github.com/nodejs/node/pull/24939
+    description: This class was renamed from `Zlib` to `ZlibBase`.
 -->
 
 Not exported by the `zlib` module. It is documented here because it is the base
@@ -440,7 +577,8 @@ Close the underlying handle.
 added: v0.5.8
 -->
 
-* `kind` **Default:** `zlib.constants.Z_FULL_FLUSH`
+* `kind` **Default:** `zlib.constants.Z_FULL_FLUSH` for zlib-based streams,
+  `zlib.constants.BROTLI_OPERATION_FLUSH` for Brotli-based streams.
 * `callback` {Function}
 
 Flush pending data. Don't call this frivolously, premature flushes negatively
@@ -460,6 +598,8 @@ added: v0.11.4
 * `strategy` {integer}
 * `callback` {Function}
 
+This function is only available for zlib-based streams, i.e. not Brotli.
+
 Dynamically update the compression level and compression strategy.
 Only applicable to deflate algorithm.
 
@@ -478,6 +618,24 @@ added: v7.0.0
 
 Provides an object enumerating Zlib-related constants.
 
+## zlib.createBrotliCompress([options])
+<!-- YAML
+added: REPLACEME
+-->
+
+* `options` {brotli options}
+
+Creates and returns a new [`BrotliCompress`][] object.
+
+## zlib.createBrotliDecompress([options])
+<!-- YAML
+added: REPLACEME
+-->
+
+* `options` {brotli options}
+
+Creates and returns a new [`BrotliDecompress`][] object.
+
 ## zlib.createDeflate([options])
 <!-- YAML
 added: v0.5.8
@@ -560,6 +718,40 @@ with `callback(error, result)`.
 Every method has a `*Sync` counterpart, which accept the same arguments, but
 without a callback.
 
+### zlib.brotliCompress(buffer[, options], callback)
+<!-- YAML
+added: REPLACEME
+-->
+* `buffer` {Buffer|TypedArray|DataView|ArrayBuffer|string}
+* `options` {brotli options}
+* `callback` {Function}
+
+### zlib.brotliCompressSync(buffer[, options])
+<!-- YAML
+added: REPLACEME
+-->
+* `buffer` {Buffer|TypedArray|DataView|ArrayBuffer|string}
+* `options` {brotli options}
+
+Compress a chunk of data with [`BrotliCompress`][].
+
+### zlib.brotliDecompress(buffer[, options], callback)
+<!-- YAML
+added: REPLACEME
+-->
+* `buffer` {Buffer|TypedArray|DataView|ArrayBuffer|string}
+* `options` {brotli options}
+* `callback` {Function}
+
+### zlib.brotliDecompressSync(buffer[, options])
+<!-- YAML
+added: REPLACEME
+-->
+* `buffer` {Buffer|TypedArray|DataView|ArrayBuffer|string}
+* `options` {brotli options}
+
+Decompress a chunk of data with [`BrotliDecompress`][].
+
 ### zlib.deflate(buffer[, options], callback)
 <!-- YAML
 added: v0.6.0
@@ -832,6 +1024,8 @@ Decompress a chunk of data with [`Unzip`][].
 [`.flush()`]: #zlib_zlib_flush_kind_callback
 [`Accept-Encoding`]: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3
 [`ArrayBuffer`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer
+[`BrotliCompress`]: #zlib_class_zlib_brotlicompress
+[`BrotliDecompress`]: #zlib_class_zlib_brotlidecompress
 [`Buffer`]: buffer.html#buffer_class_buffer
 [`Content-Encoding`]: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11
 [`DataView`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView
@@ -845,6 +1039,8 @@ Decompress a chunk of data with [`Unzip`][].
 [`Unzip`]: #zlib_class_zlib_unzip
 [`stream.Transform`]: stream.html#stream_class_stream_transform
 [`zlib.bytesWritten`]: #zlib_zlib_byteswritten
+[Brotli parameters]: #zlib_brotli_constants
 [Memory Usage Tuning]: #zlib_memory_usage_tuning
+[RFC 7932]: https://www.rfc-editor.org/rfc/rfc7932.txt
 [pool size]: cli.html#cli_uv_threadpool_size_size
 [zlib documentation]: https://zlib.net/manual.html#Constants
diff --git a/tools/doc/type-parser.js b/tools/doc/type-parser.js
@@ -38,6 +38,8 @@ const customTypesMap = {
   'AsyncHook': 'async_hooks.html#async_hooks_async_hooks_createhook_callbacks',
   'AsyncResource': 'async_hooks.html#async_hooks_class_asyncresource',
 
+  'brotli options': 'zlib.html#zlib_class_brotlioptions',
+
   'Buffer': 'buffer.html#buffer_class_buffer',
 
   'ChildProcess': 'child_process.html#child_process_class_childprocess',