Update puppeteer to version 23.4.0

[timvandermeij]

In #18771 we found that updating Puppeteer from version 23.3.0 to 23.4.0 caused new test failures. This is also reproducible locally. The logs from the bots can be found at http://54.193.163.58:8877/f9b8d1c303f789c/output.txt (Windows) and http://54.241.84.105:8877/902fcf449da1c69/output.txt (Linux). Version 23.3.1 doesn't have this issue locally, so we need to investigate what changed between 23.3.1 and 23.4.0 that causes some tests to fail and/or harden the tests against (intermittent) failure.

[timvandermeij]

I have compared the Puppeteer 23.3.1 release with the Puppeteer 23.4.0 release and found the following commits: puppeteer/puppeteer@puppeteer-v23.3.1...puppeteer-v23.4.0. There isn't really much interesting in there other than the Chrome update, and in the logs above I noticed that the failures are all in Chrome. However, the problems in Chrome might point to pre-existing issues in the tests themselves that only surface now.

[jamesgpearce]

Unrelated to pdf.js but I am also getting unusual errors from 23.4 relative to 23.3. Basic things like not being able to connect to the server for the first few tests.

[timvandermeij]

The issue exists in Puppeteer 23.5.0 too. The following logs contain the failures for me locally when running the integration tests against that version:

Puppeteer 23.5.0 integration test logs

Failures:
1) FreeText Editor Freetext must stay focused after having been moved must keep the focus
  Message:
    In chrome: Expected 0 to equal 1.
  Stack:
        at <Jasmine>
        at file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/freetext_editor_spec.mjs:2326:14
        at async Promise.all (index 1)
        at async UserContext.<anonymous> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/freetext_editor_spec.mjs:2250:7)

2) Highlight Editor Highlight with the keyboard must check that some text has been highlighted
  Message:
    ProtocolError: Waiting for selector `#pdfjs_internal_editor_0` failed: Runtime.callFunctionOn timed out. Increase the 'protocolTimeout' setting in launch/connect calls for a higher timeout if needed.
  Stack:
        at <instance_members_initializer> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:89:14)
        at new Callback (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:93:16)
        at CallbackRegistry.create (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:19:26)
        at Connection._rawSend (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:86:26)
        at CdpCDPSession.send (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:63:33)
        at #evaluate (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:355:50)
        at ExecutionContext.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:322:36)
        at IsolatedWorld.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/IsolatedWorld.js:86:30)
        at CdpJSHandle.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/api/JSHandle.js:147:37)
        at WaitTask.rerun (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/WaitTask.js:100:47)

3) Highlight Editor Highlight and caret browsing must check that selection is correctly highlighted on arrow down key pressed
  Message:
    ProtocolError: Waiting for selector `#pdfjs_internal_editor_0` failed: Runtime.callFunctionOn timed out. Increase the 'protocolTimeout' setting in launch/connect calls for a higher timeout if needed.
  Stack:
        at <instance_members_initializer> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:89:14)
        at new Callback (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:93:16)
        at CallbackRegistry.create (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:19:26)
        at Connection._rawSend (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:86:26)
        at CdpCDPSession.send (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:63:33)
        at #evaluate (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:355:50)
        at ExecutionContext.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:322:36)
        at IsolatedWorld.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/IsolatedWorld.js:86:30)
        at CdpJSHandle.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/api/JSHandle.js:147:37)
        at WaitTask.rerun (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/WaitTask.js:100:47)

4) Highlight Editor Use a toolbar overlapping an other highlight must check that the toolbar is usable
  Message:
    ProtocolError: Waiting for selector `#pdfjs_internal_editor_0` failed: Runtime.callFunctionOn timed out. Increase the 'protocolTimeout' setting in launch/connect calls for a higher timeout if needed.
  Stack:
        at <instance_members_initializer> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:89:14)
        at new Callback (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:93:16)
        at CallbackRegistry.create (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:19:26)
        at Connection._rawSend (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:86:26)
        at CdpCDPSession.send (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:63:33)
        at #evaluate (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:355:50)
        at ExecutionContext.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:322:36)
        at IsolatedWorld.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/IsolatedWorld.js:86:30)
        at CdpJSHandle.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/api/JSHandle.js:147:37)
        at WaitTask.rerun (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/WaitTask.js:100:47)

5) Ink Editor Basic operations must draw, undo/redo and check that the editor don't move
  Message:
    In chrome: Expected 420 to equal 121.
  Stack:
        at <Jasmine>
        at file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/ink_editor_spec.mjs:133:14
        at async Promise.all (index 1)
        at async UserContext.<anonymous> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/ink_editor_spec.mjs:103:7)
  Message:
    In chrome: Expected 331 to equal 32.
  Stack:
        at <Jasmine>
        at file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/ink_editor_spec.mjs:137:14
        at async Promise.all (index 1)
        at async UserContext.<anonymous> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/ink_editor_spec.mjs:103:7)

6) Interaction in issue15092.pdf must check that a value is correctly updated on a field and its siblings
  Message:
    ProtocolError: Runtime.callFunctionOn timed out. Increase the 'protocolTimeout' setting in launch/connect calls for a higher timeout if needed.
  Stack:
        at <instance_members_initializer> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:89:14)
        at new Callback (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:93:16)
        at CallbackRegistry.create (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:19:26)
        at Connection._rawSend (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:86:26)
        at CdpCDPSession.send (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:63:33)
        at #evaluate (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:355:50)
        at ExecutionContext.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:322:36)
        at IsolatedWorld.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/IsolatedWorld.js:86:30)
        at CdpJSHandle.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/api/JSHandle.js:147:37)
        at WaitTask.rerun (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/WaitTask.js:100:47)

7) Interaction in bug1802888.pdf must check field value is treated by default as a number
  Message:
    In chrome: Expected '479' to equal '579'.
  Stack:
        at <Jasmine>
        at file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/scripting_spec.mjs:1596:58
        at async Promise.all (index 1)
        at async UserContext.<anonymous> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/scripting_spec.mjs:1580:7)

8) Interaction Textfield with a zip code starting with 0 must check the zip code is correctly formatted
  Message:
    ProtocolError: Runtime.callFunctionOn timed out. Increase the 'protocolTimeout' setting in launch/connect calls for a higher timeout if needed.
  Stack:
        at <instance_members_initializer> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:89:14)
        at new Callback (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:93:16)
        at CallbackRegistry.create (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:19:26)
        at Connection._rawSend (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:86:26)
        at CdpCDPSession.send (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:63:33)
        at #evaluate (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:355:50)
        at ExecutionContext.evaluate (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:270:36)
        at IsolatedWorld.evaluate (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/IsolatedWorld.js:97:30)
        at CdpJSHandle.evaluate (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/api/JSHandle.js:139:37)
        at awaitPromise (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/test_utils.mjs:129:18)

9) Stamp Editor Basic operations must load a SVG, delete it and undo
  Message:
    ProtocolError: Runtime.callFunctionOn timed out. Increase the 'protocolTimeout' setting in launch/connect calls for a higher timeout if needed.
  Stack:
        at <instance_members_initializer> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:89:14)
        at new Callback (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:93:16)
        at CallbackRegistry.create (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:19:26)
        at Connection._rawSend (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:86:26)
        at CdpCDPSession.send (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:63:33)
        at #evaluate (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:355:50)
        at ExecutionContext.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/ExecutionContext.js:322:36)
        at IsolatedWorld.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/IsolatedWorld.js:86:30)
        at CdpJSHandle.evaluateHandle (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/api/JSHandle.js:147:37)
        at WaitTask.rerun (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/node_modules/puppeteer-core/lib/esm/puppeteer/common/WaitTask.js:100:47)

10) Text layer Text selection using mouse doesn't jump when hovering on an empty area in a single page
  Message:
    In chrome:
        Expected '' to roughly match 'code sequences, records
        them, and compiles them to fast native code. We call suc'.
  Stack:
        at <Jasmine>
        at file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/text_layer_spec.mjs:138:18
        at async Promise.all (index 1)
        at async UserContext.<anonymous> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/text_layer_spec.mjs:116:11

In contrast, running the integration test locally with the currently pinned Puppeteer 23.3.1 version gives the following much cleaners logs (the only failure here is tracked in #18774):

Puppeteer 23.3.1 integration test logs

Failures:
1) Text layer Text selection using mouse doesn't jump when hovering on an empty area in a single page
  Message:
    In chrome:
        Expected '' to roughly match 'code sequences, records
        them, and compiles them to fast native code. We call suc'.
  Stack:
        at <Jasmine>
        at file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/text_layer_spec.mjs:138:18
        at async Promise.all (index 1)
        at async UserContext.<anonymous> (file:///home/timvandermeij/Documenten/Ontwikkeling/pdf.js/Code/test/integration/text_layer_spec.mjs:116:11)

[timvandermeij]

If PR #19064 is merged we should be down to two categories of issues with the newer versions of Puppeteer:

• The Stamp Editor Basic operations must load a SVG, delete it and undo integration test, which fails pretty consistently locally and also on the bots sometimes already. This might actually be an issue in the editor code, which I have explained further in [Meta] Fix intermittent issues in the integration tests #18396 (comment) but don't actually know a good fix for yet.
• The highlight editor integration tests from Update puppeteer to version 23.4.0 #18773 (comment), which fail consistently locally but not (or barely) on the bots. This might also be an issue in the editor code or the tests that the new Puppeteer version merely surfaced: the editor never actually shows up after making the selection. I have had some luck with reproducing this by following the exact steps from the test in the viewer too, but I'm not sure on the root cause or fix here too.

/cc @calixteman If you have any ideas here that'd be great, since you're much more familiar with this code/these tests than I am. Those should be the final blockers before we can update Puppeteer again. The test failures can easily be reproduced by setting Puppeteer to version ^23.8.0 locally in package.json, running npm install and then running npx gulp integrationtest (to speed it up, only enable the highlight and stamp specs in integration-boot.js).

[timvandermeij]

The stamp editor issue is fixed by #19069, so only the highlight editor ones seem to remain.

[timvandermeij]

The highlight editor issues are (at least partially) fixed by #19111.