readline: add option to stop duplicates in history #2982

DannyNemer · 2015-09-21T14:10:33Z

Checklist

tests and code linting passes
a test and/or benchmark is included
documentation is changed or added
the commit message follows commit guidelines

Affected core subsystem(s)

readline

Description of change

Adds options.deDupeHistory for readline.createInterface(options). If options.deDupeHistory is true, when a new input line being added to the history list duplicates an older one, removes the older line from the list. Defaults to false.

To clarify, this option is for instantiating a new readline Interface; it is not for the Node REPL. The REPL inherits from Interface, but is unaffected (as is any existing Interface implementation) because this option defaults to false.

Many users would appreciate this option for creating RLIs, as it is a common setting in shells. (This setting is availble in bash, zsh, and enabled by default for fish.)

Extends documentation to reflect this API addition.

Adds tests for when options.deDupeHistory is truthy, and when options.deDupeHistory is falsey.

Fishrock123 · 2015-09-21T18:01:00Z

Seems good at a glance. Perhaps it should remove the old index and replace it with the new index?

Also, bash doesn't do this, so maybe this functionality would be unexpected?

thefourtheye · 2015-09-21T18:40:31Z

Does this mean, when I press up arrow in the REPL, it will skip the actual commands which I executed if they repeat? Is it acceptable?

DannyNemer · 2015-09-21T19:49:56Z

@Fishrock123 Bash does have this as an option, as does zsh, while fish has this enabled by default. I suggest options.deDupeHistory default tofalse for the readline Interface, just as bash and zsh do. Likewise, I think the behavior will be expected as it must be deliberately enabled.

Also, @Fishrock123, if a new line is identical to an old line, then the old line is removed from history and the new line is appended to the end of the history as normal. This is the behavior of bash, zsh, and fish when this option is enabled. Pardon me if I misunderstand your question.

@thefourtheye No, this will be not be enabled for the repl, which is why I referenced the repl as a reason options.deDupeHistory should default to false. But, if the user creates a readline Interface and sets options.deDupeHistory to true, then when a line is input that is identical to a previous line, the older line is removed from history. That is to say, when the user scrolls through their history with the up key, they will not see the same line more than once. Does this answer your question? Thank you.

Fishrock123 · 2016-06-29T12:07:52Z

doc/api/readline.md

@@ -358,6 +358,9 @@ added: v0.1.98
    only if `terminal` is set to `true` by the user or by an internal `output`
    check, otherwise the history caching mechanism is not initialized at all.
  * `prompt` - the prompt string to use. Default: `'> '`
+  * `historyNoDups` {boolean} If `true`, when a new input line being added to the


deDupeHistory may be a better name.

Fishrock123 · 2016-06-29T12:09:58Z

@DannyNemer Sorry for the extensive delay, LGTM in hindsight.

CI: https://ci.nodejs.org/job/node-test-pull-request/3122/

thefourtheye · 2016-06-29T13:35:23Z

lib/readline.js

+    if (this.deDupeHistory) {
+      // Remove older history line if identical to new one
+      const dupIndex = this.history.indexOf(this.line);
+      if (dupIndex !== -1) this.history.splice(dupIndex, 1);


I would have preferred filter over this.

Wouldn't Array.prototype.filter() be slower? The method would always create a new array whether or not a duplicate exists, and would traverse the entire this.history array even after finding a match/duplicate, though there can never be more than one duplicate.

Fishrock123 · 2016-06-30T09:00:06Z

@DannyNemer One other thing, I think it should probably also do this upon loading the history, i.e. for the internal repl's persistent history from file.

DannyNemer · 2016-06-30T15:28:15Z

@Fishrock123 I do not think so. As is, this change does not affect repl. repl inherits from Interface (to which this new option, deDupeHistory, applies), but the option defaults to false. Currently, repl has no external API for accessing the options it uses when internally instantiating its Interface instance. Hence, this current implementation does not enable the option for repl.

Were we to add this option to repl, however, it would require extending that module and the separate internal repl module. I feel a deDupeHistory option would not be useful for repl because its history contains previously evaluated code, for which a history line can be a single line from a multi-line code block entered in the repl. Ergo, removing duplicate lines might break multi-line code blocks in history.

(In contrast to repl, this deDupeHistory option is useful for command-based CLIs made with Interface. Personally, I use this option with a CLI I made that has a shell-like interface to control and query a server.)

Your thoughts? Thank you!

Fishrock123 · 2016-07-01T16:26:45Z

Makes sense to me.

Fishrock123

Looks mostly good to me.

@DannyNemer Sorry for taking so long to get to this, but if you could update it looks about good to go.

One other note is that this isn't useable via the REPL api unless you also pass the value at

node/lib/repl.js

Lines 438 to 445 in 40eba12

    
           Interface.call(this, { 
        
             input: self.inputStream, 
        
             output: self.outputStream, 
        
             completer: self.completer, 
        
             terminal: options.terminal, 
        
             historySize: options.historySize, 
        
             prompt 
        
           });

-- which would probably be preferred. Again, LGTM other than these comments.

Fishrock123 · 2016-12-16T19:40:33Z

doc/api/readline.md

@@ -365,6 +365,9 @@ added: v0.1.98
    `crlfDelay` milliseconds, both `\r` and `\n` will be treated as separate
    end-of-line input. Default to `100` milliseconds.
    `crlfDelay` will be coerced to `[100, 2000]` range.
+  * `deDupeHistory` {boolean} If `true`, when a new input line being added to the
+    history list duplicates an older one, removes the older line from the list.
+    Defaults to `false`.


"input line ~~being~~ added"

"older one, this removes the older"

Fishrock123 · 2016-12-16T19:41:30Z

lib/readline.js

@@ -48,6 +48,7 @@ function Interface(input, output, completer, terminal) {

  EventEmitter.call(this);
  var historySize;
+  var deDupeHistory;


Should this be initialized with false by default?

Done. I originally did not initialize the variable to match the style of the historySize variable declaration that precedes it, but this change is immaterial.

Fishrock123 · 2016-12-16T20:00:18Z

test/parallel/test-readline-interface.js

+  expectedLines = ['foo', 'bar', 'baz', 'bar', 'bat', 'bat'];
+  callCount = 0;
+  rli.on('line', function(line) {
+    assert.equal(line, expectedLines[callCount]);


It would probably be preferred if these assertions could all be their strict variants, e.g. assert.strictEqual() or asset.notStrictEqual().

jasnell · 2017-02-28T22:55:30Z

@DannyNemer ... is this still something you'd like to pursue?

DannyNemer · 2017-03-02T01:38:45Z

@Fishrock123 I implemented all suggested changes. Thank you for your help!

Regarding extending the repl module to support the deDupeHistory option:

One other note is that this isn't useable via the REPL api unless you also pass the value at

node/lib/repl.js

Lines 438 to 445 in 40eba12

Interface.call(this, {

input: self.inputStream,

output: self.outputStream,

completer: self.completer,

terminal: options.terminal,

historySize: options.historySize,

prompt

});

-- which would probably be preferred.

Passing the deDupeHistory parameter value at the specified location implies extending the repl module to allow instantiating an repl with the deDupeHistory option enabled (default to false). However, I feel an REPL, which only evaluates user-input JS code, should never prevent duplicates in its input history. The repl history contains previously evaluated code, for which a history line can be a single line from a multi-line code block entered in the repl. Hence, removing duplicate lines would break multi-line code blocks in history. For this reason, I feel repl should not have this option.

In contrast to repl, this deDupeHistory option is useful for command-based CLIs made with Interface (e.g., readline.createInterface()). Personally, I use this option with CLIs that I design with shell-like interfaces that control and query servers.

Your thoughts? I appreciate your suggestion.

Adds `options.deDupeHistory` for `readline.createInterface(options)`. If `options.deDupeHistory` is `true`, when a new input line being added to the history list duplicates an older one, removes the older line from the list. Defaults to `false`. Many users would appreciate this option, as it is a common setting in shells. This option certainly should not be default behavior, as it would be problematic in applications such as the `repl`, which inherits from the readline `Interface`. Extends documentation to reflect this API addition. Adds tests for when `options.deDupeHistory` is truthy, and when `options.deDupeHistory` is falsey.

Fishrock123 · 2017-03-17T19:39:26Z

My suggestion was only for the default repl, not for any new REPL(). However, I think you do have a point there.

New CI: https://ci.nodejs.org/job/node-test-pull-request/6907/

Adds `options.deDupeHistory` for `readline.createInterface(options)`. If `options.deDupeHistory` is `true`, when a new input line being added to the history list duplicates an older one, removes the older line from the list. Defaults to `false`. Many users would appreciate this option, as it is a common setting in shells. This option certainly should not be default behavior, as it would be problematic in applications such as the `repl`, which inherits from the readline `Interface`. Extends documentation to reflect this API addition. Adds tests for when `options.deDupeHistory` is truthy, and when `options.deDupeHistory` is falsey. PR-URL: #2982 Reviewed-By: Jeremiah Senkpiel <[email protected]>

Fishrock123 · 2017-03-17T21:03:24Z

Thanks, landed in 5bda5fa! Sorry for the huge delay on this one 😰.

I've merged as-is now, if we want it to be a default option for the repl, or an env variable or something we can discuss that separately. :)

Adds `options.deDupeHistory` for `readline.createInterface(options)`. If `options.deDupeHistory` is `true`, when a new input line being added to the history list duplicates an older one, removes the older line from the list. Defaults to `false`. Many users would appreciate this option, as it is a common setting in shells. This option certainly should not be default behavior, as it would be problematic in applications such as the `repl`, which inherits from the readline `Interface`. Extends documentation to reflect this API addition. Adds tests for when `options.deDupeHistory` is truthy, and when `options.deDupeHistory` is falsey. PR-URL: #2982 Reviewed-By: Jeremiah Senkpiel <[email protected]>

Renames `options.deDupeHistory` → `options.removeHistoryDuplicates` for `readline.createInterface(options)`. The option name `removeHistoryDuplicates` is preferable to the semantically identical name `deDupeHistory` because "dedupe" (short for "deduplication") is obscure and neologistic while `removeHistoryDuplicates` is clear, though verbose. Updates tests and documentation for this option accordingly. PR-URL: #11950 Ref: #2982 Reviewed-By: Vse Mozhet Byt <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Sakthipriyan Vairamani <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Evan Lucas <[email protected]> Reviewed-By: James M Snell <[email protected]>

Notable changes: * buffer: - do not segfault on out-of-range index (Timothy Gu) #11927 * crypto: - Fix memory leak if certificate is revoked (Tom Atkinson) #12089 * deps: * upgrade npm to 4.2.0 (Kat Marchán) #11389 * fix async await desugaring in V8 (Michaël Zasso) #12004 * readline: - add option to stop duplicates in history (Danny Nemer) #2982 * src: - add native URL class (James M Snell) #11801 PR-URL: #12104

Notable changes: * buffer: - do not segfault on out-of-range index (Timothy Gu) nodejs/node#11927 * crypto: - Fix memory leak if certificate is revoked (Tom Atkinson) nodejs/node#12089 * deps: * upgrade npm to 4.2.0 (Kat Marchán) nodejs/node#11389 * fix async await desugaring in V8 (Michaël Zasso) nodejs/node#12004 * readline: - add option to stop duplicates in history (Danny Nemer) nodejs/node#2982 * src: - add native URL class (James M Snell) nodejs/node#11801 PR-URL: nodejs/node#12104 Signed-off-by: Ilkka Myller <[email protected]>

MylesBorins · 2017-05-08T15:30:44Z

@nodejs/lts we should discuss this backport as it has been requested by @bnoordhuis

jasnell · 2017-05-08T15:38:16Z

+1 on landing in 6.x

MylesBorins · 2017-05-15T15:36:49Z

needs to land with #11950

Adds `options.deDupeHistory` for `readline.createInterface(options)`. If `options.deDupeHistory` is `true`, when a new input line being added to the history list duplicates an older one, removes the older line from the list. Defaults to `false`. Many users would appreciate this option, as it is a common setting in shells. This option certainly should not be default behavior, as it would be problematic in applications such as the `repl`, which inherits from the readline `Interface`. Extends documentation to reflect this API addition. Adds tests for when `options.deDupeHistory` is truthy, and when `options.deDupeHistory` is falsey. PR-URL: #2982 Reviewed-By: Jeremiah Senkpiel <[email protected]>

Renames `options.deDupeHistory` → `options.removeHistoryDuplicates` for `readline.createInterface(options)`. The option name `removeHistoryDuplicates` is preferable to the semantically identical name `deDupeHistory` because "dedupe" (short for "deduplication") is obscure and neologistic while `removeHistoryDuplicates` is clear, though verbose. Updates tests and documentation for this option accordingly. PR-URL: #11950 Ref: #2982 Reviewed-By: Vse Mozhet Byt <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Sakthipriyan Vairamani <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Evan Lucas <[email protected]> Reviewed-By: James M Snell <[email protected]>

Adds `options.deDupeHistory` for `readline.createInterface(options)`. If `options.deDupeHistory` is `true`, when a new input line being added to the history list duplicates an older one, removes the older line from the list. Defaults to `false`. Many users would appreciate this option, as it is a common setting in shells. This option certainly should not be default behavior, as it would be problematic in applications such as the `repl`, which inherits from the readline `Interface`. Extends documentation to reflect this API addition. Adds tests for when `options.deDupeHistory` is truthy, and when `options.deDupeHistory` is falsey. PR-URL: #2982 Reviewed-By: Jeremiah Senkpiel <[email protected]>

Renames `options.deDupeHistory` → `options.removeHistoryDuplicates` for `readline.createInterface(options)`. The option name `removeHistoryDuplicates` is preferable to the semantically identical name `deDupeHistory` because "dedupe" (short for "deduplication") is obscure and neologistic while `removeHistoryDuplicates` is clear, though verbose. Updates tests and documentation for this option accordingly. PR-URL: #11950 Ref: #2982 Reviewed-By: Vse Mozhet Byt <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Sakthipriyan Vairamani <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Evan Lucas <[email protected]> Reviewed-By: James M Snell <[email protected]>

This LTS release comes with 126 commits. This includes 40 which are test related, 32 which are doc related, 12 which are build / tool related and 4 commits which are updates to dependencies. Notable Changes: * build: - support for building mips64el (nanxiongchao) #10991 * cluster: - disconnect() now returns a reference to the disconnected worker. (Sean Villars) #10019 * crypto: - ability to select cert store at runtime (Adam Majer) #8334 - Use system CAs instead of using bundled ones (Adam Majer) #8334 - The `Decipher` methods `setAuthTag()` and `setAAD` now return `this`. (Kirill Fomichev) #9398 - adding support for OPENSSL_CONF again (Sam Roberts) #11006 - make LazyTransform compabile with Streams1 (Matteo Collina) #12380 * deps: - upgrade libuv to 1.11.0 (cjihrig) #11094 - upgrade libuv to 1.10.2 (cjihrig) #10717 - upgrade libuv to 1.10.1 (cjihrig) #9647 - upgrade libuv to 1.10.0 (cjihrig) #9267 * dns: - Implemented `{ttl: true}` for `resolve4()` and `resolve6()` (Ben Noordhuis) #9296 * process: - add NODE_NO_WARNINGS environment variable (cjihrig) #10842 * readline: - add option to stop duplicates in history (Danny Nemer) #2982 * src: - support "--" after "-e" as end-of-options (John Barboza) #10651 * tls: - new tls.TLSSocket() supports sec ctx options (Sam Roberts) #11005 - Allow obvious key/passphrase combinations. (Sam Roberts) #10294 PR-URL: #13059

Adds `options.deDupeHistory` for `readline.createInterface(options)`. If `options.deDupeHistory` is `true`, when a new input line being added to the history list duplicates an older one, removes the older line from the list. Defaults to `false`. Many users would appreciate this option, as it is a common setting in shells. This option certainly should not be default behavior, as it would be problematic in applications such as the `repl`, which inherits from the readline `Interface`. Extends documentation to reflect this API addition. Adds tests for when `options.deDupeHistory` is truthy, and when `options.deDupeHistory` is falsey. PR-URL: nodejs/node#2982 Reviewed-By: Jeremiah Senkpiel <[email protected]>

Renames `options.deDupeHistory` → `options.removeHistoryDuplicates` for `readline.createInterface(options)`. The option name `removeHistoryDuplicates` is preferable to the semantically identical name `deDupeHistory` because "dedupe" (short for "deduplication") is obscure and neologistic while `removeHistoryDuplicates` is clear, though verbose. Updates tests and documentation for this option accordingly. PR-URL: nodejs/node#11950 Ref: nodejs/node#2982 Reviewed-By: Vse Mozhet Byt <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Sakthipriyan Vairamani <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Evan Lucas <[email protected]> Reviewed-By: James M Snell <[email protected]>

This LTS release comes with 126 commits. This includes 40 which are test related, 32 which are doc related, 12 which are build / tool related and 4 commits which are updates to dependencies. Notable Changes: * build: - support for building mips64el (nanxiongchao) nodejs/node#10991 * cluster: - disconnect() now returns a reference to the disconnected worker. (Sean Villars) nodejs/node#10019 * crypto: - ability to select cert store at runtime (Adam Majer) nodejs/node#8334 - Use system CAs instead of using bundled ones (Adam Majer) nodejs/node#8334 - The `Decipher` methods `setAuthTag()` and `setAAD` now return `this`. (Kirill Fomichev) nodejs/node#9398 - adding support for OPENSSL_CONF again (Sam Roberts) nodejs/node#11006 - make LazyTransform compabile with Streams1 (Matteo Collina) nodejs/node#12380 * deps: - upgrade libuv to 1.11.0 (cjihrig) nodejs/node#11094 - upgrade libuv to 1.10.2 (cjihrig) nodejs/node#10717 - upgrade libuv to 1.10.1 (cjihrig) nodejs/node#9647 - upgrade libuv to 1.10.0 (cjihrig) nodejs/node#9267 * dns: - Implemented `{ttl: true}` for `resolve4()` and `resolve6()` (Ben Noordhuis) nodejs/node#9296 * process: - add NODE_NO_WARNINGS environment variable (cjihrig) nodejs/node#10842 * readline: - add option to stop duplicates in history (Danny Nemer) nodejs/node#2982 * src: - support "--" after "-e" as end-of-options (John Barboza) nodejs/node#10651 * tls: - new tls.TLSSocket() supports sec ctx options (Sam Roberts) nodejs/node#11005 - Allow obvious key/passphrase combinations. (Sam Roberts) nodejs/node#10294 PR-URL: nodejs/node#13059

brendanashworth added the readline Issues and PRs related to the built-in readline module. label Sep 21, 2015

Trott force-pushed the master branch from 1e896a6 to 082cc8d Compare December 27, 2015 02:00

estliberitas force-pushed the master branch 2 times, most recently from 7da4fd4 to c7066fb Compare April 26, 2016 05:22

Fishrock123 reviewed Jun 29, 2016
View reviewed changes

Fishrock123 added the semver-minor PRs that contain new features and should be released in the next minor version. label Jun 29, 2016

Fishrock123 self-assigned this Jun 29, 2016

thefourtheye reviewed Jun 29, 2016
View reviewed changes

Trott force-pushed the master branch from b0df363 to c5ce7f4 Compare September 21, 2016 00:09

rvagg force-pushed the master branch 2 times, most recently from c133999 to 83c7a88 Compare October 18, 2016 17:01

jasnell added the stalled Issues and PRs that are stalled. label Oct 28, 2016

Fishrock123 suggested changes Dec 16, 2016

View reviewed changes

MylesBorins force-pushed the master branch from 8df7ee0 to 54fef67 Compare February 1, 2017 01:00

Fishrock123 approved these changes Mar 17, 2017

View reviewed changes

Fishrock123 closed this Mar 17, 2017

Fishrock123 mentioned this pull request Mar 17, 2017

Add $HOME/.config/nodejs with an option for deduping default repl history #11907

Closed

Fishrock123 removed the dont-land-on-v7.x label Mar 23, 2017

MylesBorins mentioned this pull request Mar 28, 2017

V7.8.0 proposal #12104

Merged

italoacasas mentioned this pull request Apr 10, 2017

v7.9.0 Release Proposal #12319

Merged

2 tasks

gibfahn mentioned this pull request Apr 24, 2017

Potential Semver Minor Backports nodejs/Release#188

Closed

31 tasks

MylesBorins added the lts-watch-v6.x label May 8, 2017

MylesBorins added the lts-agenda label May 8, 2017

MylesBorins added land-on-v6.x and removed lts-watch-v6.x labels May 16, 2017

MylesBorins mentioned this pull request May 23, 2017

v6.11.0 proposal #13059

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

readline: add option to stop duplicates in history #2982

readline: add option to stop duplicates in history #2982

DannyNemer commented Sep 21, 2015 •

edited

Loading

Fishrock123 commented Sep 21, 2015

thefourtheye commented Sep 21, 2015

DannyNemer commented Sep 21, 2015 •

edited

Loading

Fishrock123 Jun 29, 2016

DannyNemer Jun 29, 2016

Fishrock123 commented Jun 29, 2016

thefourtheye Jun 29, 2016

DannyNemer Jun 29, 2016

Fishrock123 commented Jun 30, 2016 •

edited

Loading

DannyNemer commented Jun 30, 2016 •

edited

Loading

Fishrock123 commented Jul 1, 2016

Fishrock123 left a comment

Fishrock123 Dec 16, 2016

DannyNemer Mar 2, 2017

Fishrock123 Dec 16, 2016

DannyNemer Mar 2, 2017

Fishrock123 Dec 16, 2016

DannyNemer Mar 2, 2017

jasnell commented Feb 28, 2017

DannyNemer commented Mar 2, 2017 •

edited

Loading

Fishrock123 commented Mar 17, 2017

Fishrock123 commented Mar 17, 2017

MylesBorins commented May 8, 2017

jasnell commented May 8, 2017

MylesBorins commented May 15, 2017

	Interface.call(this, {
	input: self.inputStream,
	output: self.outputStream,
	completer: self.completer,
	terminal: options.terminal,
	historySize: options.historySize,
	prompt
	});

readline: add option to stop duplicates in history #2982

readline: add option to stop duplicates in history #2982

Conversation

DannyNemer commented Sep 21, 2015 • edited Loading

Checklist

Affected core subsystem(s)

Description of change

Fishrock123 commented Sep 21, 2015

thefourtheye commented Sep 21, 2015

DannyNemer commented Sep 21, 2015 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fishrock123 commented Jun 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fishrock123 commented Jun 30, 2016 • edited Loading

DannyNemer commented Jun 30, 2016 • edited Loading

Fishrock123 commented Jul 1, 2016

Fishrock123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented Feb 28, 2017

DannyNemer commented Mar 2, 2017 • edited Loading

Fishrock123 commented Mar 17, 2017

Fishrock123 commented Mar 17, 2017

MylesBorins commented May 8, 2017

jasnell commented May 8, 2017

MylesBorins commented May 15, 2017

DannyNemer commented Sep 21, 2015 •

edited

Loading

DannyNemer commented Sep 21, 2015 •

edited

Loading

Fishrock123 commented Jun 30, 2016 •

edited

Loading

DannyNemer commented Jun 30, 2016 •

edited

Loading

DannyNemer commented Mar 2, 2017 •

edited

Loading