Auto-instrumentation - never ending pod mutation

[mat-rumian]

Environment:

• OpenTelemetry-Operator deployed 0.52.0
• some other operator which is modifying environment variables of the pod container

Error Flow:

1. OT Operator adds environment variables, volume and init container with auto-instrumentation
2. other operator takes the same pod and adds some other environment variables
3. OT Operator detects changes in the pod and try to update it (env vars)
4. other operator again detect changes and change env vars
5. and so on...

Final error log from k8s:

Failed to annotate pod with branch ENI details: Pod "test-pod-64cf7b8d5b-j8zbv" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations) 

Expected flow for OT Operator:

1. OT Operator checks (before pod mutation) if instrumentation is configured for pod
2. OT Operator adds environment variables, volume and init container with auto-instrumentation
3. other operator add applies his stuff
4. OT Operator doesn't make any other changes as pod is already instrumented

[mat-rumian]

@pavolloffay I will be happy to help with this as already investigated the issue. I'm not sure about solution, how should we check if pod is already mutated by OT Operator:

1. check if env vars are already configured?
2. check if init container is already configured?
3. introduce additional annotation to the pod and check if exists (add annotation to the pod after instrumentation configuration part)
4. something else?