No Violations found after enabling audit-from-cache

[laxmanvallandas]

What steps did you take and what happened:
We have set of opa policies defined and all were validated without using audit-from-cache as we did not had the requirement to validate based on Pre-existed data.

All objects which do not respect policy were denied/warned and same was visible in audit report of respective constraint definition.
Sample:

kubectl get k8srequiredlabels.constraints.gatekeeper.sh all-must-have-owner -oyaml
....
  totalViolations: 3
  violations:
  - enforcementAction: warn
    kind: Namespace
    message: All namespaces must have an `owner` label that points to your company
      username
    name: production

I have recently enabled audit-from-cache to true and defined a new constrainttemplates which validates if a specific annotation already exists across all services in the cluster.
Config:

spec:
  sync:
    syncOnly:
    - group: ""
      kind: Service
      version: v1

With above config and defining(not shared consdering not relevant here, can provide if needed) new constratintemplate uniqueannotation, was able to successfully restrict the object creation when any service attempts to go against the policy.
Also, It does capture the violations on the new constraint that I defined.

However, the audit report of existing constraints do not capture the results on the violations anymore. But, it still restricts the creation of objects which are not as per polices.

Please note that I have enabled Sync only on Service kind because I do not want to use memory by filling audit cache of all other k8s objects(which I do not validate using PRE-EXISTING data as before in existing constrainttemplates).

I see that similar bug was reported before #2026
But, that seems to be about not seeing violation on the newly created constraint template, which in my case is working.

What did you expect to happen:

With the issue stated, opa gatekeeper still restricts(warn/deny) the objects during creation time, but it audit report which used to help me in the past about on which object has attempted to go against the policy is no longer possible now.

Expected: Constraints should still contain violation report info though audit from cache is enabled.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Gatekeeper version: 3.11.0
• Kubernetes version: (use kubectl version): 1.24.11

[davis-haba]

What do you mean by "pre-existing data"? Why are you interesting in using audit-from-cache?

The main reason to use audit-from-cache is to limit the numbers of requests audit sends to the API server. Audit-from-cache simply tells audit to use the Informer cache, instead of asking the API server for all resources under audit.

If this is your use case and it's not working as expected, can you please share an example of a Service you are expecting to be rejected, along with the ConstraintTemplate and Constraints you are using to validate the Service.

[laxmanvallandas]

Thanks for looking into it.

What do you mean by "pre-existing data"? Why are you interesting in using audit-from-cache?

I mean validating based on audit cache(data.inventory). We want to validate if any of the new "Service" that we create would NOT be allowed if any of the existing "Service" in the cluster has same key:value pair in its annotation already.

The main reason to use audit-from-cache is to limit the numbers of requests audit sends to the API server. Audit-from-cache simply tells audit to use the Informer cache, instead of asking the API server for all resources under audit.

Absolutely. That was clear.

If this is your use case and it's not working as expected, can you please share an example of a Service you are expecting to be rejected, along with the ConstraintTemplate and Constraints you are using to validate the Service.

I think the issue i am facing is misunderstood. Apologies, if that was not clear.

Here it goes:
Before audit-from-cache is enabled:

1. We had several constrainttemplates and constraints. Let's take an example of one constraint on imagepullpolicy. If any pod doesn't have imagepullpolicy set to always, it will be denied.
2. Let's say a pod has denied it, which helps the user to update the imagepullpolicy.
3. ALSO, same has been captured in Violations count.
   Sample:

~:kube-system [~]$ kubectl get imagepullpolicyconstraint
NAME                                     ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
image-pull-policy-for-my-namespaces      deny                 1

4. ALSO, below was helpful in understanding which pod has denied the constraint
   Sample:

:kube-system [~]$ kubectl get imagepullpolicyconstraint image-pull-policy-for-my-namespace -o  jsonpath\={.status.violations[*].name}
mypod-6c88dd56c9-vj5qm

After audit-from-cache is enabled and with config as shared before:

1. Deployed new constrainttemplates:
   Snippet:

        hostnames = split(input.review.object.metadata.annotations[input.parameters.annotation], ",")

        violation[{"msg": msg}] {
          hostname := hostnames[_]
          h = trim(hostname, " ")
          svc_objs := [o | o = data.inventory.namespace[_][_][input.parameters.kind][_]]
          all_values := {h | obj = svc_objs[_]; h = obj.metadata.annotations[input.parameters.annotation]}
          count({h} - all_values) == 0
          msg := sprintf("annotation <%v> with value <%v> already exists for another %v", [input.parameters.annotation, h, input.review.object.kind])
        }

2. Deployed new Constraint:
   Snippet:

spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Service"]
  parameters:
    annotation: <custom annotation key>
    kind: Service

3. With above, as stated earlier, when a service is created with annotation(key:value) which is already used by another service across all namespace, Service created would be denies by gatekeeper.
4. Please note, it works as expected.
5. Now, If I try to create a pod with unexpected imagepullpolicy(See Before audit-from-cache is enabled) , pod creation is denied(AS EXPECTED and there is no issue here), HOWEVER Total Violations is not incremented

~:kube-system [~]$ kubectl get imagepullpolicyconstraint
NAME                                     ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
image-pull-policy-for-my-namespaces      deny                 0

6. And, describing constraint doesn't share details of pods that violated the policy ANYMORE. This is feature which is broke for me after enabling audit-from-cache.
   Sample:

:kube-system [~]$ kubectl get imagepullpolicyconstraint image-pull-policy-for-my-namespace -o  jsonpath\={.status.violations[*].name}

7. BUT, describing constraint that is using audit-cache(data.inventory), it does list the objects which violated them. All good here.
   Sample:

:kube-system [~]$ kubectl get uniqueannotation service-should-not-have-same-annotation -o  jsonpath\={.status.violations[*].name}
myservicename => which violated the constraint

To conclude, though gatekeeper acting as per enforcement policy but the Issue is, we are missing(POINT 6 Above) the information on the list of objects that violated the constraints(which was not the case before enabling audit-from-cache).
Let me know if that helps and correct me if there is some misunderstanding in my expectation.

[ritazh]

Thanks for sharing the details. So seems the issue is with the imagepullpolicyconstraint constraint, and the unique service annotation is working fine.

For the imagepullpolicyconstraint constraint, it’s a deny policy, can you please confirm that you have at lease one pod violating this currently running in the cluster? ie not all requests have been denied.

[laxmanvallandas]

Yes, all constraints working fine to either deny/warn. Total Violations information of the constraint is missing after audit-from-cache is enabled.

For the imagepullpolicyconstraint constraint, it’s a deny policy, can you please confirm that you have at lease one pod violating this currently running in the cluster? ie not all requests have been denied.

Yes, I can confirm. imagepullpolicyconstraint is just an example, we have few other constraints and run tests(has pods which covers policies which are violated and few are not) to capture the violated object names.

[maxsmythe]

From your config, it looks like you are not syncing pods:

spec:
  sync:
    syncOnly:
    - group: ""
      kind: Service
      version: v1

If you want to see audit violations for pods and you want to use the --audit-from-cache flag, then you will need to sync pods (and any other resource you want to have audited).

To be clear, you do not need the --audit-from-cache flag for constraint templates that use data.inventory to work. Instructing Gatekeeper to sync the necessary resources using the syncOnly config option is sufficient.

[laxmanvallandas]

Enabling --audit-from-cache is definitely not what we are looking for as we have constraints on all objects including pods, jobs etc.

To be clear, you do not need the --audit-from-cache flag for constraint templates that use data.inventory to work. Instructing Gatekeeper to sync the necessary resources using the syncOnly config option is sufficient.

Ah. That's the miss understanding from my side. Thanks.