Error of admission webhook prevents scaling of deployment despite falurePolicy: Ignore

[trutty]

We use auto-instrumentation via annotation with our pods. For that, there has to be an Instrumentation CRD in the same namespace as the pod. If this is not the case the following error occurs in the opentelemetry-operator:

msg":"failed to select an OpenTelemetry Instrumentation instance for this pod","namespace":"","name":"","error":"no OpenTelemetry Instrumentation instances available"

The error message is fine, but what happens is that the pod/deployment does not get scaled up because the addmission webhook denies the request with the event:

message: 'Error creating: admission webhook "mpod.kb.io" denied the request: no OpenTelemetry Instrumentation instances available'

because of

opentelemetry-operator/internal/webhookhandler/webhookhandler.go

Lines 85 to 88 in ecb139c

	pod, err = m.Mutate(ctx, ns, pod)
	if err != nil {
	return admission.Errored(http.StatusInternalServerError, err)
	}

From my understanding, this error should be ignored and the respective deployment/pod (which has the auto-instumentation annotation) should be scaled up nevertheless, albeit without the mutation, because the admission webhook has failurePolicy: Ignore. Am I missing something?

The instrumentation feature is not critical for the deployment (at least in my context), so I would rather have a running pod without instrumentation than no pod at all.

[pavolloffay]

You are right, we shouldn't be blocking the deployment I think that should be the default behavior.

If there are folks that would like to fail the deployment we could add it as a config on the operator.

[pavolloffay]

Adding collector label as the issue should be present for the collector as well.