fix: Only apply execution control to nodes that are not part of exit handler.

[jswxstw]

Fixes #13060
Fixes #13252

Motivation

1. This PR feat: Support argo plugin stop. Fixes #12333 #12441 has support plugin nodes shutdown, but HTTP node still not.

2. Fixes _ controller sending many pod delete requests that result in 404 response #12659 Split to PR: fix: Only cleanup agent pod if exists. Fixes #12659 #13294

3. Fixes onExit steps executed after deadline exceeded are stopped if an onExit pod spends >=10 seconds in Pending state #13060 Fixes workflow deadline is not ignored for exit handler #13252

Modifications

Modification 1

Execution control has been applied to the nodes with created pods after pod reconciliation. However, pending and suspended nodes do not have created pods, and taskset nodes use the agent pod. Mark these nodes failed when shutting down or exceeding deadline since pod reconciliation does not take effect on them.

Modification 2

Agent pod will only be created when taskset nodes exist, so do not delete it if it does not exist.
Split to PR: #13294

Modification 3

Only apply execution control for nodes that are not part of exit handler:

• failNodesWithoutCreatedPodsAfterDeadlineOrShutdown: Pending nodes will not be failed if they are part of exit handler.
• createWorkflowPod: Do not set ARGO_DEADLINE if it is onExitPod.

Verification

local test and e2e tests

[JasonChen86899]

make some comments in pr(#13185) about IsExitNode function
Or maybe need to be clear about the definition of an ExitNode

1. The root node of the exit handle
2. Any one of all the nodes involved in the exit handle

[jswxstw]

Or maybe need to be clear about the definition of an ExitNode

I think we can do it this way: keep the existing logic of IsExitNode and add a new function IsPartOfExitHandler to determine if the node is in the Exit Handler.

[JasonChen86899]

// IsPartOfExitHandler returns whether node is part of exit handler.
func (woc *wfOperationCtx) IsPartOfExitHandler(node wfv1.NodeStatus) bool {
	if node.BoundaryID == "" {
		return node.IsExitNode()
	}
	if boundaryNode, err := woc.wf.Status.Nodes.Get(node.BoundaryID); err == nil {
		return boundaryNode.IsExitNode()
	}
	return false
}

A for loop or recursion is needed here. Here's a modification of my code based on yours

// IsPartOfExitHandler returns whether node is part of exit handler.
func (woc *wfOperationCtx) IsPartOfExitHandler(node wfv1.NodeStatus) (bool, error) {
	tmpNode := node
	for !tmpNode.IsExitNode() {
		if tmpNode.BoundaryID == "" {
			return false, nil
		}
		// get boundary node
		boundaryNode, err := woc.wf.Status.Nodes.Get(tmpNode.BoundaryID)
		if err != nil {
			return false, err
		}

		tmpNode = *boundaryNode
	}
	
	return true, nil
}

[agilgur5]

Modification 2

Agent pod will only be created when taskset nodes exist, so do not delete it if it does not exist.

@jswxstw can you split the agent delete into a separate PR? it's independent of the other changes here and supersedes #12679, which we both already reviewed, so that one I can review and merge almost immediately. (It's also a lot simpler/has a lot less edge cases to review than the rest)
Can assign me on that split PR

[jswxstw]

New PR: #13294 @agilgur5, I don't have permission to assign.

[agilgur5]

New PR: #13294

Thanks! Approved. Should merge once tests succeed.

I don't have permission to assign.

Ah right. For future reference, I think you can "Request a Review" which will at least send me a notification

[terrytangyuan]

Thanks!