High memory consumption

[adamhenson]

Thank you for your work on this project. This issue seems similar to #9633, but it doesn't seem like there was a concrete answer or solution. I've been experiencing high levels of memory consumption on Lighthouse audits for a while now (Lighthouse 4 - 6), some URLs worse than others, typically around 2GiB per run, and sometimes up to 6GiB. The issue for me was more pronounced on one specific URL which would consistently go above the memory limit of my pod and cause it to be evicted... pretty much sabotaging my entire stack and in the end I had to upgrade my AWS EC2 instances to t3.xlarge 💸

I believe a memory consumption that sometimes reaches 6GiB for a single audit is a critical problem. Are there settings I could be missing? I'm happy to help in any way possible.

Reproduction

Environment

• OS: Linux x86 (64 bit)
• Node Version: 12.18.3
• Lighthouse Version: 6.3.0 (but I don't think this is specific to version)
• Chrome Launcher Version: 0.13.2 (but I don't think this is specific to version)

Example URL

• https://www.kitchenaid.com.br/

Configuration

I'm basically using the default configuration as defined here.

Over the years my options have evolved (or devolved) to this. I had reason for setting each option, but I've forgotten why at this point:

const options = {
  chromeFlags: [
    '--disable-dev-shm-usage',
    '--headless',
    '--no-sandbox',
    '--ignore-certificate-errors',
  ],

  // two minutes
  maxWaitForLoad: 120000,
  port: LIGHTHOUSE_API_CHROME_PORT,
};

Current Behavior

I've set a resource memory limit on my pod of 5GiB and once in a while the pod becomes evicted, because the memory limit is exceeded (and the pod is shut down and restarted). I've logged the output of a run on the URL noted above. The bottom of the log is the point where Kubernetes shuts it down.

Expand here to see log

Thu, 27 Aug 2020 23:29:41 GMT config:warn IFrameElements gatherer requested, however no audit requires it.
Thu, 27 Aug 2020 23:29:41 GMT config:warn FormElements gatherer requested, however no audit requires it.
Thu, 27 Aug 2020 23:29:41 GMT status Connecting to browser
Thu, 27 Aug 2020 23:29:41 GMT status Resetting state with about:blank
Thu, 27 Aug 2020 23:29:41 GMT status Benchmarking machine
Thu, 27 Aug 2020 23:29:42 GMT status Initializing…
Thu, 27 Aug 2020 23:29:42 GMT status Running defaultPass pass CSSUsage, JsUsage, ViewportDimensions, RuntimeExceptions, ConsoleMessages, AnchorElements, ImageElements, LinkElements, MetaElements, ScriptElements, IFrameElements, FormElements, MainDocumentContent, GlobalListeners, AppCacheManifest, Doctype, DOMStats, OptimizedImages, PasswordInputsWithPreventedPaste, ResponseCompression, TagsBlockingFirstPaint, FontSize, EmbeddedContent, RobotsTxt, TapTargets, Accessibility, TraceElements, InspectorIssues, SourceMaps
Thu, 27 Aug 2020 23:29:42 GMT status Resetting state with about:blank
Thu, 27 Aug 2020 23:29:42 GMT status Setting up network for the pass trace
Thu, 27 Aug 2020 23:29:42 GMT status Cleaning browser cache
Thu, 27 Aug 2020 23:29:42 GMT status Beginning devtoolsLog and trace
Thu, 27 Aug 2020 23:29:42 GMT status Loading page & waiting for onload
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: CSSUsage
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: JsUsage
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: ViewportDimensions
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: RuntimeExceptions
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: ConsoleMessages
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: AnchorElements
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: ImageElements
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: LinkElements
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: MetaElements
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: ScriptElements
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: IFrameElements
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: FormElements
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: MainDocumentContent
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: GlobalListeners
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: AppCacheManifest
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: Doctype
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: DOMStats
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: OptimizedImages
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: PasswordInputsWithPreventedPaste
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: ResponseCompression
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: TagsBlockingFirstPaint
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: FontSize
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: EmbeddedContent
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: RobotsTxt
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: TapTargets
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: Accessibility
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: TraceElements
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: InspectorIssues
Thu, 27 Aug 2020 23:29:49 GMT status Gathering in-page: SourceMaps
Thu, 27 Aug 2020 23:29:49 GMT status Gathering trace
Thu, 27 Aug 2020 23:29:50 GMT status Gathering devtoolsLog & network records
Thu, 27 Aug 2020 23:29:50 GMT status Gathering: CSSUsage
Thu, 27 Aug 2020 23:29:51 GMT status Gathering: JsUsage
Thu, 27 Aug 2020 23:29:51 GMT status Gathering: ViewportDimensions
Thu, 27 Aug 2020 23:29:51 GMT status Gathering: RuntimeExceptions
Thu, 27 Aug 2020 23:29:51 GMT status Gathering: ConsoleMessages
Thu, 27 Aug 2020 23:29:51 GMT status Gathering: AnchorElements
Thu, 27 Aug 2020 23:29:51 GMT status Gathering: ImageElements
Thu, 27 Aug 2020 23:29:51 GMT status Gathering: LinkElements
Thu, 27 Aug 2020 23:29:51 GMT status Gathering: MetaElements
Thu, 27 Aug 2020 23:29:51 GMT status Gathering: ScriptElements
Thu, 27 Aug 2020 23:29:52 GMT method <= browser ERR:error Network.getResponseBody 
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: IFrameElements
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: FormElements
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: MainDocumentContent
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: GlobalListeners
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: AppCacheManifest
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: Doctype
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: DOMStats
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: OptimizedImages
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: PasswordInputsWithPreventedPaste
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: ResponseCompression
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: TagsBlockingFirstPaint
Thu, 27 Aug 2020 23:29:52 GMT status Gathering: FontSize

Expected Behavior

I would expect memory consumption to always be lower than 2GiB.

Related issues
#9633

[patrickhulce]

Thanks for filing @adamhenson! #11289 is also very related which is the other place where we've observed OOMs from the same style of audit.

I can't see any unusual memory footprint on my machine but I can reproduce a crash on 100% of runs on this URL and filed https://bugs.chromium.org/p/chromium/issues/detail?id=1123027

[adamhenson]

Thanks @patrickhulce.