util: improve util.inspect performance #14492

BridgeAR · 2017-07-26T04:25:59Z

A hole in a sparse array is now in itself a O(1) operation instead of O(n) where n is the number of coherent elements of the hole.

In addition a hasOwnProperty check was replaced with a cheaper one.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

util

This is a huge performance improvement in case of sparse arrays when using util.inspect as the hole will simple be skipped.

tniessen · 2017-07-26T11:29:42Z

CI: https://ci.nodejs.org/job/node-test-pull-request/9354/

mscdex · 2017-07-26T17:03:32Z

lib/util.js

-      index++;
+      index = i;
+      visibleLength++;
+      if (visibleLength === ctx.maxArrayLength)


Couldn't these previous two lines be merged into if (++visibleLength === ctx.maxArrayLength) ?

TimothyGu · 2017-07-26T17:16:20Z

lib/util.js

+      break;
+    const i = +elem;
+    if (index !== i) {
+      if (i > 0 === false)


Can you add a comment on why this is necessary, and different from i <= 0? (I understand it because I saw an earlier version of the PR, but comments are more for future than now anyway.)

BridgeAR · 2017-07-27T03:44:26Z

Comments addressed

aqrln · 2017-07-28T03:14:33Z

Thanks for addressing the issue! I made some benchmarks back when it was opened, so feel free to include them in your pull request if you wish: https://gist.github.com/aqrln/195743d38f632b5b9a8d5944f30cb6af

Speaking of which, I observe a performance regression with dense arrays:

                                                improvement confidence      p.value
 util/inspect-array.js len=100000 n=1000           -35.07 %        *** 1.862234e-35
 util/inspect-sparse-array.js len=100000 n=1000   2662.66 %        *** 1.516381e-41

Can you reproduce it or is it just me (the machine was a bit loaded with other stuff, so the result might be unreliable)?

aqrln · 2017-07-28T03:30:07Z

lib/util.js

@@ -803,7 +813,7 @@ function formatProperty(ctx, value, recurseTimes, visibleKeys, key, array) {
      str = ctx.stylize('[Setter]', 'special');
    }
  }
-  if (!hasOwnProperty(visibleKeys, key)) {
+  if (visibleKeys[key] === undefined) {


It may be worth to eventually make visibleKeys an ES2015 Set instead of emulating it via an object, but I don't know how it will affect performance with the current V8 (if it will at all).

When I looked at the code there were a couple more optimizations possible but I tried to be more surgical here. E.g. for arrays there is no need for Object.keys when using for in and ctx.seen should be a Set instead of an Array.

BridgeAR · 2017-07-28T13:35:41Z

@aqrln thanks for the benchmarks. I saw some minor drop as well and changed it now. With for in there was also a minor regression (a edge case) and I added a test for that.

Current benchmark:

 util/inspect-array.js type="denseArray" len=100000 n=100       1.09 %            1.853220e-01
 util/inspect-array.js type="mixedArray" len=100000 n=100    2596.93 %        *** 8.927819e-18
 util/inspect-array.js type="sparseArray" len=100000 n=100   3455.27 %        *** 1.563057e-17

aqrln · 2017-07-28T13:49:47Z

CI: https://ci.nodejs.org/job/node-test-pull-request/9392/

BridgeAR · 2017-08-08T05:05:01Z

PTAL

refack · 2017-08-13T17:53:38Z

Landed in 95bbb68
Extra sanity test of master: https://ci.nodejs.org/job/node-test-commit-linuxone/7938/ ✔️

* improve util.inspect performance This is a huge performance improvement in case of sparse arrays when using util.inspect as the hole will simple be skipped. * use faster visibleKeys property lookup * add inspect-array benchmark PR-URL: #14492 Fixes: #14487 Reviewed-By: Alexey Orlenko <[email protected]> Reviewed-By: Refael Ackermann <[email protected]>

MylesBorins · 2017-09-20T02:54:32Z

I'm not sure if we want to backport this to v6.x, but I've put it under baking

BridgeAR · 2017-09-20T03:04:52Z

@MylesBorins I think it should be backported - especially as there are other commits that should also be backported at some point and they depend on this. But when this is backported #14880 has to be backported as well due to the minor regression introduced here.

BridgeAR · 2017-12-11T06:19:02Z

I think enough time has passed to backport this. From this change itself there was no further complain about any regression. What do you think @MylesBorins ?

BridgeAR added 2 commits July 26, 2017 01:17

util: use for in instead of while hasOwnProperty

ddd4de4

This is a huge performance improvement in case of sparse arrays when using util.inspect as the hole will simple be skipped.

util: use faster visibleKeys property lookup

b73a399

nodejs-github-bot added the util Issues and PRs related to the built-in util module. label Jul 26, 2017

fixup

ab497df

mscdex reviewed Jul 26, 2017

View reviewed changes

mscdex added the performance Issues and PRs related to the performance of Node.js. label Jul 26, 2017

TimothyGu reviewed Jul 26, 2017

View reviewed changes

fixup comments commit 1

61119e4

aqrln reviewed Jul 28, 2017

View reviewed changes

BridgeAR added 2 commits July 28, 2017 10:08

fixup commit 1

a2574f0

benchmarks: add inspect-array benchmark

4aed280

aqrln approved these changes Jul 28, 2017

View reviewed changes

jasnell requested a review from Fishrock123 August 8, 2017 23:38

BridgeAR mentioned this pull request Aug 11, 2017

util: fix circular ref for Set and Map #14775

Closed

4 tasks

refack approved these changes Aug 13, 2017

View reviewed changes

refack self-assigned this Aug 13, 2017

refack closed this Aug 13, 2017

addaleax mentioned this pull request Aug 13, 2017

v8.4.0 proposal #14811

Merged

BridgeAR added the backported-to-v8.x label Sep 14, 2017

MylesBorins added the baking-for-lts PRs that need to wait before landing in a LTS release. label Sep 20, 2017

MylesBorins added the lts-watch-v6.x label Sep 20, 2017

BridgeAR mentioned this pull request Dec 20, 2017

doc: warn about using util.inspect/util.format in prod #17791

Closed

3 tasks

MylesBorins removed the baking-for-lts PRs that need to wait before landing in a LTS release. label Aug 17, 2018

refack removed their assignment Oct 12, 2018

BridgeAR deleted the fix-util-format branch April 1, 2019 23:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

util: improve util.inspect performance #14492

util: improve util.inspect performance #14492

BridgeAR commented Jul 26, 2017

tniessen commented Jul 26, 2017

mscdex Jul 26, 2017

BridgeAR Jul 27, 2017

TimothyGu Jul 26, 2017

BridgeAR Jul 27, 2017

BridgeAR commented Jul 27, 2017

aqrln commented Jul 28, 2017

aqrln Jul 28, 2017

BridgeAR Jul 28, 2017 •

edited

Loading

BridgeAR commented Jul 28, 2017 •

edited

Loading

aqrln commented Jul 28, 2017

BridgeAR commented Aug 8, 2017

refack commented Aug 13, 2017 •

edited

Loading

MylesBorins commented Sep 20, 2017

BridgeAR commented Sep 20, 2017

BridgeAR commented Dec 11, 2017

util: improve util.inspect performance #14492

util: improve util.inspect performance #14492

Conversation

BridgeAR commented Jul 26, 2017

Checklist

Affected core subsystem(s)

tniessen commented Jul 26, 2017

mscdex Jul 26, 2017

Choose a reason for hiding this comment

BridgeAR Jul 27, 2017

Choose a reason for hiding this comment

TimothyGu Jul 26, 2017

Choose a reason for hiding this comment

BridgeAR Jul 27, 2017

Choose a reason for hiding this comment

BridgeAR commented Jul 27, 2017

aqrln commented Jul 28, 2017

aqrln Jul 28, 2017

Choose a reason for hiding this comment

BridgeAR Jul 28, 2017 • edited Loading

Choose a reason for hiding this comment

BridgeAR commented Jul 28, 2017 • edited Loading

aqrln commented Jul 28, 2017

BridgeAR commented Aug 8, 2017

refack commented Aug 13, 2017 • edited Loading

MylesBorins commented Sep 20, 2017

BridgeAR commented Sep 20, 2017

BridgeAR commented Dec 11, 2017

BridgeAR Jul 28, 2017 •

edited

Loading

BridgeAR commented Jul 28, 2017 •

edited

Loading

refack commented Aug 13, 2017 •

edited

Loading