[BUG] ArrayNode - fatal error: concurrent map writes

[sonjaer]

Describe the bug

Propeller (v1.11.0) started crashing/restarting with:

{"data":{"exec_id":"f0b816c75f7f94134b6d","ns":"commerce-alerts-pipeline-development","res_ver":"10125211633","routine":"worker-287","src":"executor.go:372","wf":"commerce-alerts-pipeline:development:commerce_alerts_pipeline.workflow.main_workflow"},"message":"Handling Workflow [f0b816c75f7f94134b6d], id: [project:\"commerce-alerts-pipeline\"  domain:\"development\"  name:\"f0b816c75f7f94134b6d\"], p [Running]","severity":"INFO","timestamp":"2024-04-03T09:00:47Z"}
fatal error: concurrent map writes

goroutine 539 [running]:
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/array.newArrayExecutionContext({0x32f70a8, 0xc27fb3e780}, 0x4a09a40?)
	/go/pkg/mod/github.com/flyteorg/flyte/[email protected]/pkg/controller/nodes/array/execution_context.go:29 +0x94
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/array.newArrayNodeExecutionContext({0x32f1860, 0xc27f80d200}, {0x32d70b0?, 0xc27fb4def0}, {0x32dea00?, 0xc27fb6c720}, 0xc280f9fd70?, 0xc280fa1880)
	/go/pkg/mod/github.com/flyteorg/flyte/[email protected]/pkg/controller/nodes/array/node_execution_context.go:110 +0x65
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/array.(*arrayNodeExecutionContextBuilder).BuildNodeExecutionContext(0xc280f959a0, {0x32d9fa8?, 0xc280f9f830?}, {0x32f70a8?, 0xc27fb3e780?}, {0x32db4f0?, 0xc280f9fd70?}, {0xc280fac218, 0x2})
	/go/pkg/mod/github.com/flyteorg/flyte/[email protected]/pkg/controller/nodes/array/node_execution_context_builder.go:32 +0xd1
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).RecursiveNodeHandler(0xc280f959f0, {0x32d9fa8, 0xc280f9f830}, {0x32f70a8, 0xc27fb3e780}, {0x32c3850, 0xc280f9fd70}, {0x32db4f0?, 0xc280f9fd70?}, {0x32f4290, ...})
	/go/pkg/mod/github.com/flyteorg/flyte/[email protected]/pkg/controller/nodes/executor.go:208 +0x4a3
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/array.(*worker).run.func1(0xc1a77f3f50, 0xc001792f00?, 0xc1a77f3f30)
	/go/pkg/mod/github.com/flyteorg/flyte/[email protected]/pkg/controller/nodes/array/worker.go:68 +0xd3
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/array.(*worker).run(0xc000cb4cc0)
	/go/pkg/mod/github.com/flyteorg/flyte/[email protected]/pkg/controller/nodes/array/worker.go:70 +0x145
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/array.(*arrayNodeHandler).Setup.func1()
	/go/pkg/mod/github.com/flyteorg/flyte/[email protected]/pkg/controller/nodes/array/handler.go:570 +0x17
created by github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/array.(*arrayNodeHandler).Setup in goroutine 482
	/go/pkg/mod/github.com/flyteorg/flyte/[email protected]/pkg/controller/nodes/array/handler.go:569 +0x25

We think it's due to a bug with ArrayNode. The specific workflow (commerce-alerts-pipeline:development:commerce_alerts_pipeline.workflow.main_workflow) referenced before stacktrace has this code and when we stop running it propeller seems to stop restarting

from flytekit.experimental import map_task

Expected behavior

Running this task won't crash propeller

Additional context to reproduce

Let me know if you want more info

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes