Skip to content

Latest commit

 

History

History
430 lines (333 loc) · 15.2 KB

maintaining-dependencies.md

File metadata and controls

430 lines (333 loc) · 15.2 KB
Raw

Maintaining Dependencies

Node.js depends on additional components beyond the Node.js code itself. These dependencies provide both native and JavaScript code and are built together with the code under the src and lib directories to create the Node.js binaries.

All dependencies are located within the deps directory. This a list of all the dependencies:

Any code which meets one or more of these conditions should be managed as a dependency:

  • originates in an upstream project and is maintained in that upstream project.
  • is not built from the preferred form of the work for making modifications to it (see GNU GPL v2, section 3. when make node is run. A good example is WASM code generated from C (the preferred form). Typically generation is only supported on a subset of platforms, needs additional tools, and is pre-built outside of the make node step and then committed as a WASM binary in the directory for the dependency under the deps directory.

By default all dependencies are bundled into the Node.js binary, however, configure options should be available to use an externalized version at runtime when:

  • the dependency provides native code and is available as a shared library in one or more of the common Node.js distributions.
  • the dependency provides JavaScript and is not built from the preferred form of the work for making modifications to it when make node is run.

Many distributions use externalized dependencies for one or more of these reasons:

  1. They have a requirement to build everything that they ship from the preferred form of the work for making modifications to it. This means that they need to replace any pre-built components (for example WASM binaries) with an equivalent that they have built.
  2. They manage the dependency separately as it is used by more applications than just Node.js. Linking against a shared library allows them to manage updates and CVE fixes against the library instead of having to patch all of the individual applications.
  3. They have a system wide configuration for the dependency that all applications should respect.

Supporting externalized dependencies with native code

Support for externalized dependencies with native code for which a shared library is available can added by:

  • adding options to configure.py. These are added to the shared_optgroup and include an options to:
    • enable use of a shared library
    • set the name of the shared library
    • set the path to the directory with the includes for the shared library
    • set the path to where to find the shared library at runtime
  • add a call to configure_library() to configure.py for the library at the end of list of existing configure_library() calls. If there are additional libraries that are required it is possible to list more than one with the pkgname option.
  • in node.gypi guard the build for the dependency with node_shared_depname so that it is only built if the dependency is being bundled into Node.js itself. For example:
    [ 'node_shared_brotli=="false"', {
      'dependencies': [ 'deps/brotli/brotli.gyp:brotli' ],
    }],

Supporting externalizable dependencies with JavaScript code

Support for an externalizable dependency with JavaScript code can be added by:

  • adding an entry to the shareable_builtins map in configure.py. The path should correspond to the file within the deps directory that is normally bundled into Node.js. For example deps/cjs-module-lexer/lexer.js. This will add a new option for building with that dependency externalized. After adding the entry you can see the new option by running ./configure --help.

  • adding a call to AddExternalizedBuiltin to the constructor for BuiltinLoader in src/node_builtins.cc for the dependency using the NODE_SHARED_BUILTLIN #define generated for the dependency. After running ./configure with the new option you can find the #define in config.gypi. You can cut and paste one of the existing entries and then update to match the import name for the dependency and the #define generated.

  • if the version of the dependency is reported in process.versions, update src/node_metadata.h and src/node_metadata.cc so that the version is not reported when the dependency is externalized. Not reporting the version is better than incorrectly reporting the version of the dependency bundled with Node.js, instead of the version for the externalized dependency. Use one of the existing externalized dependencies, like Undici, as an example of how to update these files correctly. Make sure to run the tests with the dependency externalized, as the tests will also need to be updated to handle this properly.

Supporting non-externalized dependencies with JavaScript code

If the dependency consists of JavaScript in the preferred form of the work for making modifications to it, it can be added as a non-externalizable dependency. In this case simply add the path to the JavaScript file in the deps_files list in the node.gyp file.

Common approach for dependencies with WASM components

WASM components within dependencies are most often built outside of the regular Node.js make build step. They also require different tools.

It is important that the tools and their versions used to build WASM components shipped within Node.js are well documented and be available if needed to rebuild/update older Node.js versions.

In order to minimize the different number of tools and versions used to build WASM components and to document and ensure future availability, the project builds and maintains a common wasm-builder container that should be use to build WASM components in Node.js dependencies.

The container provides a durable copy of the versions of the tools used for a specific build which are under the control of the Node.js project. In addition, the tools and verions are documented through metadata within the container in the /home/node/metadata directory.

The available tools can be found by looking at the current version of the Dockerfile used to create the container.

If additional WASM tool are needed beyond those available in the container, additions should be PR'd into the wasm-builder container.

Examples of using the container include:

In addition to using the container to build WASM components, the goal is also for the WASM components and final files that are shipped with Node.js to be built by the dep-updaters that are run on a regular basis and that they use only the files available in the Node.js repo for the dependency. For example, being able to rebuild the WASM and files that we ship in Node.js using only the files in ../deps/undici.

Updating dependencies

Most dependencies are automatically updated by dependency-update-action that runs weekly. However, it is possible to manually update a dependency by running the corresponding script in tools/update-deps. OpenSSL has its own update action: update-openssl-action. npm-cli-bot takes care of npm update, it is maintained by the npm team.

PRs for manual dependency updates should only be accepted if the update cannot be generated by the automated tooling, the reason is clearly documented and either the PR is reviewed in detail or it is from an existing collaborator.

In general updates to dependencies should only be accepted if they have already landed in the upstream. The TSC may grant an exception on a case-by-case basis. This avoids the project having to float patches for a long time and ensures that tooling can generate updates automatically.

Dependency list

acorn

The acorn dependency is a JavaScript parser. acorn-walk is an abstract syntax tree walker for the ESTree format.

ada

The ada dependency is a fast and spec-compliant URL parser written in C++.

amaro

The amaro dependency is a wrapper around the WebAssembly version of the SWC JavaScript/TypeScript parser.

brotli

The brotli dependency is used for the homonym generic-purpose lossless compression algorithm.

c-ares

The c-ares is a C library for asynchronous DNS requests.

cjs-module-lexer

The cjs-module-lexer dependency is used within the Node.js ESM implementation to detect the named exports of a CommonJS module. See maintaining-cjs-module-lexer for more information.

corepack

The corepack dependency is a zero-runtime-dependency Node.js script that acts as a bridge between Node.js projects and the package managers they are intended to be used with during development. In practical terms, Corepack will let you use Yarn and pnpm without having to install them - just like what currently happens with npm, which is shipped by Node.js by default.

googletest

The googletest dependency is Google’s C++ testing and mocking framework.

histogram

The histogram dependency is a C port of High Dynamic Range (HDR) Histogram.

ic

The icu is widely used set of C/C++ and Java libraries providing Unicode and Globalization support for software applications. See maintaining-icu for more information.

libuv

The libuv dependency is a multi-platform support library with a focus on asynchronous I/O. It was primarily developed for use by Node.js.

llhttp

The llhttp dependency is the http parser used by Node.js. See maintaining-http for more information.

minimatch

The minimatch dependency is a minimal matching utility.

nghttp2

The nghttp2 dependency is a C library implementing HTTP/2 protocol. See maintaining-http for more information.

nghttp3

The nghttp3 dependency is HTTP/3 library written in C. See ngtcp2 for more information.

ngtcp2

The ngtcp2 and nghttp3 dependencies provide the core functionality for QUIC and HTTP/3.

The sources are pulled from:

In both the ngtcp2 and nghttp3 git repos, the active development occurs in the default branch (currently named main in each). Tagged versions do not always point to the default branch.

We only use a subset of the sources for each.

The nghttp3 library depends on ngtcp2. Both should always be updated together. From ngtcp2 we only want the contents of the lib and crypto directories; from nghttp3 we only want the contents of the lib directory.

npm

The npm dependency is the package manager for JavaScript.

New pull requests should be opened when a "next" version of npm has been released. Once the "next" version has been promoted to "latest" the PR should be updated as necessary.

The specific Node.js release streams the new version will be able to land into are at the discretion of the release and LTS teams.

This process only covers full updates to new versions of npm. Cherry-picked changes can be reviewed and landed via the normal consensus seeking process.

openssl

The openssl dependency is a fork of OpenSSL to enable QUIC. OpenSSL is toolkit for general-purpose cryptography and secure communication.

Node.js currently uses the quictls/openssl fork, which closely tracks the main openssl/openssl releases with the addition of APIs to support the QUIC protocol. See maintaining-openssl for more information.

postject

The postject dependency is used for the Single Executable strategic initiative.

simdjson

The simdjson dependency is a C++ library for fast JSON parsing.

simdutf

The simdutf dependency is a C++ library for fast UTF-8 decoding and encoding.

sqlite

The sqlite dependency is an embedded SQL database engine written in C.

undici

The undici dependency is an HTTP/1.1 client, written from scratch for Node.js.. See maintaining-http for more information.

uvwasi

The uvwasi dependency implements the WASI system call API, so that WebAssembly runtimes can easily implement WASI calls. Under the hood, uvwasi leverages libuv where possible for maximum portability. See maintaining-web-assembly for more information.

V8

V8 is Google's open source high-performance JavaScript and WebAssembly engine, written in C++. See maintaining-V8 for more information.

zlib

The zlib dependency lossless data-compression library, it comes from the Chromium team's zlib fork which incorporated performance improvements not currently available in standard zlib.