Arm32: System.Threading.Tasks.Dataflow.Tests failing with NRE

[jkotas]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=140623
Build error leg or test failing: System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
Pull request: #80323

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "System.Threading.Tasks.ConcurrentExclusiveSchedulerPair.ProcessConcurrentTasks",
  "BuildRetry": false
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[ghost]

Tagging subscribers to this area: @dotnet/area-system-threading-tasks
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=140623
Build error leg or test failing: System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
Pull request: #80323

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "System.Threading.Tasks.ConcurrentExclusiveSchedulerPair.ProcessConcurrentTasks",
  "BuildRetry": false
}

Author:	jkotas
Assignees:	-
Labels:	arch-arm32, area-System.Threading.Tasks, blocking-clean-ci, untriaged, Known Build Error
Milestone:	-

[stephentoub]

According to the stack trace, the NRE is occurring here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ConcurrentExclusiveSchedulerPair.cs

Lines 422 to 441 in 31e4f40

	for (int i = 0; i < m_maxItemsPerTask; i++)
	{
	// Get the next available concurrent task. If we can't find one, bail.
	if (!m_concurrentTaskScheduler.m_tasks.TryDequeue(out Task? concurrentTask)) break;
	// Execute the task. If the scheduler was previously faulted,
	// this task could have been faulted when it was queued; ignore such tasks.
	if (!concurrentTask.IsFaulted) m_concurrentTaskScheduler.ExecuteTask(concurrentTask);
	// Now check to see if exclusive tasks have arrived; if any have, they take priority
	// so we'll bail out here. Note that we could have checked this condition
	// in the for loop's condition, but that could lead to extra overhead
	// in the case where a concurrent task arrives, this task is launched, and then
	// before entering the loop an exclusive task arrives. If we didn't execute at
	// least one task, we would have spent all of the overhead to launch a
	// task but with none of the benefit. There's of course also an inherent
	// race condition here with regards to exclusive tasks arriving, and we're ok with
	// executing one more concurrent task than we should before giving priority to exclusive tasks.
	if (!m_exclusiveTaskScheduler.m_tasks.IsEmpty) break;
	}

on the line:

if (!concurrentTask.IsFaulted) m_concurrentTaskScheduler.ExecuteTask(concurrentTask); 

The only state being read here are readonly fields initialized in the ctor or the concurrentTask that's read from the queue. The queue is either a ConcurrentQueue<T> or a:
https://github.com/dotnet/runtime/blob/main/src/libraries/Common/src/System/Collections/Concurrent/SingleProducerSingleConsumerQueue.cs
It's not clear from the stack trace what exact test is failing and thus which queue is in use, but my money is on there either being a bug around the use (or lack thereof) of volatile accesses in the SPSCQ or a codegen bug on arm32, resulting in the concurrentTask dequeued from the queue being read as null.

[VSadov]

failed in #84151

[ericstj]

codegen bug on arm32, resulting in the concurrentTask dequeued from the queue being read as null.

Given we have zero recent hits of this I'm leaning towards this as the cause. Closing this as no-repo. Reactivate if the failures show up again.