Cluster Operator dynamic logging configuration #3328

sknot-rh · 2020-07-16T12:08:19Z

Type of change

Enhancement / new feature

Description

The CO is deployed as a Deployment so it is not possible just change envvar without rolling update. We need to use an external confingmap (maybe we should allow to configure its name).
Whenever this configmap is updated, the volume is updated and thanks to monitorInterval entry in the log4j2.properties the configuration is reloaded.

Checklist

Write tests
Make sure all tests pass
Update documentation
Check RBAC rights for Kubernetes / OpenShift roles
Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
Reference relevant issue(s) and close them after merging
Update CHANGELOG.md
Supply screenshots for visual changes, such as Grafana dashboards

...-operator/src/main/resources/config-maps/050-ConfigMap-strimzi-cluster-operator-logging.yaml

cluster-operator/src/main/resources/log4j2.properties

...s/helm2/strimzi-kafka-operator/templates/050-ConfigMap-strimzi-cluster-operator-logging.yaml

helm-charts/helm2/Makefile

helm-charts/helm3/Makefile

...s/helm3/strimzi-kafka-operator/templates/050-ConfigMap-strimzi-cluster-operator-logging.yaml

install/cluster-operator/050-ConfigMap-strimzi-cluster-operator-logging.yaml

install/cluster-operator/051-Deployment-strimzi-cluster-operator.yaml

scholzj · 2020-07-16T21:57:40Z

systemtest/src/main/java/io/strimzi/systemtest/resources/operator/BundleResource.java

@@ -20,7 +20,7 @@
 public class BundleResource {
    private static final Logger LOGGER = LogManager.getLogger(BundleResource.class);

-    public static final String PATH_TO_CO_CONFIG = "../install/cluster-operator/050-Deployment-strimzi-cluster-operator.yaml";
+    public static final String PATH_TO_CO_CONFIG = "../install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml";


@Frawless Will these renamings play well with upgrade tests which use different versions?

Interesting question, in upgrade tests, we change this file only for upgrade to HEAD, for older versions we basically do kubectl apply -f /tmp/upgrade-54556445/strimzi-0.x.y/install/cluster-operator and for HEAD Standa did proper changes from my POV. Let's try to run upgrade tests and we will see.

Frawless · 2020-07-17T09:03:09Z

@strimzi-ci run tests profile=upgrade testcase=StrimziUpgradeST#testUpgradeStrimziVersion

samuel-hawker · 2020-07-17T12:39:03Z

install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml

@@ -17,6 +17,10 @@ spec:
        strimzi.io/kind: cluster-operator


Why are we switching 50 to 60? Are none of the numbers between 40-50 appropriate for the new CM?

04x are for the CRDs. The numbers are basically just an aid for listing files. We need to have ConfigMap file before Deployment. Switching Deployment to 060 makes some space to any future ConfigMap. Maybe @scholzj has more comments.

I'd also argue 51 would also be appropriate for the CM (keeping deployment as 50) as the CM should not need to be created prior to the deployment due to the dynamic file watcher?

Well, we used different numbers for different types of resources so far. This really just continues.

strimzi-ci · 2020-07-17T12:47:06Z

✔️ Test Summary ✔️

TEST_PROFILE: upgrade
EXCLUDED_GROUPS: networkpolicies,flaky
TEST_CASE: StrimziUpgradeST#testUpgradeStrimziVersion
TOTAL: 1
PASS: 1
FAIL: 0
SKIP: 0
BUILD_NUMBER: 1226
BUILD_ENV: oc cluster up

install/cluster-operator/050-ConfigMap-strimzi-cluster-operator-logging.yaml

...s/helm3/strimzi-kafka-operator/templates/050-ConfigMap-strimzi-cluster-operator-logging.yaml

helm-charts/helm2/strimzi-kafka-operator/values.yaml

helm-charts/helm2/strimzi-kafka-operator/templates/060-Deployment-strimzi-cluster-operator.yaml

samuel-hawker · 2020-07-17T12:57:06Z

cluster-operator/scripts/cluster_operator_run.sh

+
+if [ -f /opt/strimzi/custom-config/log4j2.properties ]; then
+    # if ConfigMap was not mounted and thus this file was not created, use properties file from the classpath
+    export JAVA_OPTS="${JAVA_OPTS} -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties"


Should we not instead always deploy the CM and pre-seed the CM with the properties from the properties file?
Easier to predict behaviour if we don't have this if check here, I can imagine us having to suggets users restart operator pod to make the operator run through this logic again.

We are deploying it always. @scholzj thinks there may be some edge cases when the CM is not deployed/mounted as a volume. In that case, we should use at least some defaults (what is done by loading log4j2.properties from classpath).

Can we echo something to say that a static logging configuration is being used because no config was found, and therefore that dynamic log config update won't work.

cluster-operator/src/test/resources/log4j2-test.properties

scholzj

The code looks good to me. But I think we need to have also some docs:

You should remove STRIMZI_LOG_LEVEL from the docs
You should probably add some small chapter about the log configuration into https://strimzi.io/docs/operators/latest/full/using.html#assembly-operators-cluster-operator-str or something like that.

CHANGELOG.md

sknot-rh · 2020-07-23T15:56:37Z

documentation/modules/ref-operator-cluster.adoc

@@ -121,6 +118,31 @@ The DNS domain name is added to the Kafka broker certificates used for hostname
 +
 If you are using a different DNS domain name suffix in your cluster, change the `KUBERNETES_SERVICE_DNS_DOMAIN` environment variable from the default to the one you are using in order to establish a connection with the Kafka brokers.

+== Logging configuration


@PaulRMellor @laidan6000
Could you please check these changes?

@stanlyDoge 👍

scholzj

Some nits to the docs. But looks mostly good.

documentation/modules/ref-operator-cluster.adoc

scholzj · 2020-07-23T21:49:10Z

CHANGELOG.md

@@ -22,6 +22,7 @@
  * enable/disable metrics in the KafkaBridge custom resource
  * new Grafana dashboard for the bridge metrics
 * Support dynamically changeable logging in the Entity Operator and Kafka Bridge 
+* Configure Cluster Operator logging using ConfigMap instead of environment variable and support dynamic changes


FYI: Looks like this will not get into 0.19.0, so you might need to move this to 0.20.0.

@stanlyDoge I still see this here in 0.19.0, should you remove it as asked by Jakub?

Signed-off-by: Stanislav Knot <[email protected]>

ppatierno · 2020-08-04T13:23:51Z

install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml

@@ -17,6 +17,10 @@ spec:
        strimzi.io/kind: cluster-operator
    spec:
      serviceAccountName: strimzi-cluster-operator
+      volumes:
+        - name: co-logging-volume


even in this case the volume could be about co-configuration not just logging?

ppatierno · 2020-08-04T13:37:39Z

helm-charts/helm2/strimzi-kafka-operator/values.yaml

@@ -9,7 +9,8 @@ image:
  repository: strimzi
  name: operator
  tag: latest
-logLevel: INFO
+logVolume: cluster-operator-logging-volume


should be co-config-volume as for the Helm 3 values YAML.

Signed-off-by: Stanislav Knot <[email protected]>

tombentley · 2020-08-04T16:33:36Z

cluster-operator/scripts/cluster_operator_run.sh

+
+if [ -f /opt/strimzi/custom-config/log4j2.properties ]; then
+    # if ConfigMap was not mounted and thus this file was not created, use properties file from the classpath
+    export JAVA_OPTS="${JAVA_OPTS} -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties"


Can we echo something to say that a static logging configuration is being used because no config was found, and therefore that dynamic log config update won't work.

documentation/modules/ref-operator-cluster.adoc

helm-charts/helm2/strimzi-kafka-operator/templates/050-ConfigMap-strimzi-cluster-operator.yaml

Signed-off-by: Stanislav Knot <[email protected]>

sknot-rh · 2020-08-07T09:20:21Z

cluster-operator/scripts/cluster_operator_run.sh

+    # if ConfigMap was not mounted and thus this file was not created, use properties file from the classpath
+    export JAVA_OPTS="${JAVA_OPTS} -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties"
+else
+    echo "Configuration file log4j2.properties not found. Using default static logging setting. Dynamic updates of logging configuration will not work."


@laidan6000 Can you please review this message?

* co dyn log Signed-off-by: Stanislav Knot <[email protected]> * rename files Signed-off-by: Stanislav Knot <[email protected]> * delete file Signed-off-by: Stanislav Knot <[email protected]> * default conf Signed-off-by: Stanislav Knot <[email protected]> * commit Signed-off-by: Stanislav Knot <[email protected]> * remove redundant vars Signed-off-by: Stanislav Knot <[email protected]> * Sam's comments Signed-off-by: Stanislav Knot <[email protected]> * consistency Signed-off-by: Stanislav Knot <[email protected]> * comment Signed-off-by: Stanislav Knot <[email protected]> * docu Signed-off-by: Stanislav Knot <[email protected]> * docomments Signed-off-by: Stanislav Knot <[email protected]> * rebase + move to 0.20.0 Signed-off-by: Stanislav Knot <[email protected]> * addressing comments Signed-off-by: Stanislav Knot <[email protected]> * avoid logging Signed-off-by: Stanislav Knot <[email protected]> * another comment Signed-off-by: Stanislav Knot <[email protected]> * Tom's comments Signed-off-by: Stanislav Knot <[email protected]>

sknot-rh requested review from Frawless, ppatierno, scholzj and tombentley July 16, 2020 12:08

sknot-rh added this to the 0.19.0 milestone Jul 16, 2020

scholzj reviewed Jul 16, 2020

View reviewed changes

sknot-rh force-pushed the co-dyn-log branch from 6d6fd84 to 4213fd6 Compare July 17, 2020 09:39