[BUG] page.goto / page.waitForNavigation: networkidle resolves too early

[DigitalFlow]

Context:

• Playwright Version: 1.6.2
• Operating System: Windows 10
• Node.js version: 15.3.0
• Browser: Chromium
• Extra: -

Code Snippet

Help us help you! Put down a short code snippet that illustrates your bug and
that we can run and debug locally. For example:

const {chromium, webkit, firefox} = require('playwright');

(async () => {
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  await Promise.all([
            page.goto(`[crmUr]/main.aspx?etn=[entityName]&pagetype=entityrecord&appid=[AppId]`, { waitUntil: "load", timeout: 60000 }),
            page.waitForNavigation({ waitUntil: "networkidle", timeout: 60000 })
        ]);
})();

Describe the bug

Hey everyone,

thanks for providing this library.
I'm writing an UI Test Framework for Dynamics 365, which I recently migrated from puppeteer to playwright.
The conversion was straight forward, but I'm noticing that the networkidle option for page.waitForNavigation and page.goto resolves way too early.

Dynamics365 loads lots of data when navigating, but waiting for networkidle and load still resolves prematurely.
page.waitForFunction is not that easy, because lots of different data has to be fetched.
networkidle0 on puppeteer has worked well for this usecase, but networkidle on playwright unfortunately resolves way too early.

I found the snippet that @dguzman posted, but it seems to resolve prematurely as well, even when using longPolls = 0.

This might be a similar issue as #2515, is there anything we can do about improving networkidle?

Thanks for your help.

Kind Regards,
Florian

[aslushnikov]

Hey @DigitalFlow,

In principle there should be no functional difference with networkidle in Playwright and networkidle0 in Puppeteer. We'd be happy to look into this if you can provide us with a URL that manifests the issue.

In general, however, networkidle is an unfortunate signal to rely upon in web automation: network idleness is an unreliable concept in browsers, and you might end up with scripts that don't work consistently. I'd recommend use some other signal, ideally some DOM element or the likes.

[DigitalFlow]

Hi @aslushnikov,

thanks for your response.

I would be happy to help, unfortunately you can use Dynamics 365 only with a valid login / organization.
You can create a free trial at trials.dynamics.com, but you probably don't want to do that?

Could I otherwise also help with screencaptures and logs?
I fear that waiting for UI is not completely sufficent in dynamics, as custom scripts can be attached to forms and I don't know of a generic way of checking whether all are loaded without relying on open / closed connections.

You are right that networkidle is sometimes unreliable due to polling requests, telemetry and so on.
For this cause I was able to find all logging / telemetry endpoints that are used in dynamics and to put them on a blocklist, where I abort all requests using your page.route function.

How can we proceed with this?

Kind Regards,
Florian

[DigitalFlow]

Hi @aslushnikov,

according to what you said, I tried getting rid of networkidle wait operations.
As it turns out, thankfully Microsoft got our back and implemented a global function in Dynamics 365, which can tell us whether the page is idle or not.
I am now using this function instead of all networkidle tests and test runs have become much more stable.

I'll close this issue, thanks for your support and this library, it already solved a few issues that caused me headaches with puppeteer :)

Kind Regards,
Florian