Roslyn failing to clean up listeners

[runfoapp]

Runfo Tracking Issue: Roslyn failing to clean up listeners

Build	Definition	Kind	Run Name

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

[sharwell]

This problem will go away as soon as either #44514 or #48662 is merged. I would consider #44514 to be the "true" fix for it.

[jaredpar]

One of those PRs has been open for five months, the other six days. If we can't merge a fix today then we need to push foward with disabling the assert. This is the number 1 blocker for our builds. It's been a known issue for sometime and hasn't gotten the necessary attention.

[jaredpar]

Having a little issue with the live stats function but you can see the damage here.

https://runfo.azurewebsites.net/search/tests/?bq=definition%3Aroslyn-ci++started%3A%7E7&tq=message%3A%22Failed+to+clean+up+listeners%22

[sharwell]

Historically there were many other causes for this. Here are some of the other PRs that helped:

#48077
#47995
#47994

It was difficult to demonstrate the need for #44514 when more acute problems remained, but now that it's clear and the complex ConfiguredYieldAwaitable was reviewed as a separate change, it's set to auto-merge and should be merged today.

[jaredpar]

The live issue is now fixed so stats are updating. The data only goes back ~15 days for error messages though. I didn't start tracking that until yesterday and only ran a back fill push for last 200 or so builds.

[jaredpar]

@sharwell the PR you listed has merged, the error still persists. This is still our #1 failure in Roslyn. It's holding back all our other work.

[jaredpar]

Part of the issue here appears to be that the code is using a one minute timeout to gate cancellation. The IDE team also made a change to CI to over utilize cores on all of our test legs. That isn't a reasonable combination. The timeout can and will get hit for normal starvation reasons on a regular basis. We need to pull this timeout entirely if we want to have stable CI

[sharwell]

Part of the issue here appears to be that the code is using a one minute timeout to gate cancellation

Tests are expected to cancel all pending operations before the end of the test. The cleanup code has fast-path handling for this case to guarantee the timeout will not come into play for correctly written code. All failures of this check are either a true product bug or a true test bug for the test that timed out.

Some known product bugs remain. For example, #44522 is a case where we fail to cancel asynchronous operations when the owning object is disposed. We just haven't reached a point where we could demonstrate that particular code path was responsible for a failure because the investigations keep revealing other cases that we fix.

[jaredpar]

So how do we move forward here? This is the primary issue causing instability in CI.

My instinct is to say that we should delete the assert until we get fixes for the underlyin gproduct bugs.

[sharwell]

The most recent two changes that are likely to reduce occurrences of this are #48662 (c8eecdb) and #44514 (0b6ff9b). I need a list of failures which included both of these changes to investigate what remains.

[CyrusNajmabadi]

Closing out due to lack of movement in 4 years. IF we need to do something here, we shoudl do it.