Inconsistent configmap multiline string output between kustomize versions

[mleklund]

Inconsistent multiline string handling between versions of kustomize make it difficult to see the real changes that are being applied. I am not sure if there are "rules" that kustomize is using in the yaml output and where they might be published, but the large change-sets this introduces are hard to reconcile, and give pause to operators when there has not been any actual changes.

@stefanprodan mentions this in #3559 here where his is told that it is a separate issue.

Here is a set of files I was able to reproduce with using the example grafana.ini file:

grafana1.txt
grafana2.txt
kustomization.yaml.txt

The issue seems to surround overall length of files used in configmaps, but I have not been able to figure out exactly what is going on.

kustomize 3.5.4 output

apiVersion: v1
data:
  grafana1.txt: |
    ##################### Grafana Configuration Defaults #####################
    #
    # Do not modify this file in grafana installs
    #
    # possible values : production, development
    app_mode = production
    # instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
    instance_name = ${HOSTNAME}
    #################################### Paths ###############################
    [paths]
    # Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
    data = data
    # Temporary files in `data` directory older than given duration will be removed
    temp_data_lifetime = 24h
    # Directory where grafana can store logs
    logs = data/log
    # Directory where grafana will automatically scan and look for plugins
    plugins = data/plugins
    # folder that contains provisioning config files that grafana will apply on startup and while running.
    provisioning = conf/provisioning
    #################################### Server ##############################
    [server]
    # Protocol (http, https, h2, socket)
    protocol = http
    # The ip address to bind to, empty will bind to all interfaces
    http_addr =
    # The http port to use
    http_port = 3000
    # The public facing domain name used to access grafana from a browser
    domain = localhost
    # Redirect to correct domain if host header does not match domain
    # Prevents DNS rebinding attacks
    enforce_domain = false
    # The full public facing url
    root_url = %(protocol)s://%(domain)s:%(http_port)s/
    # Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
    serve_from_sub_path = false
    # Log web requests
    router_logging = false
    # the path relative working path
    static_root_path = public
    # enable gzip
    enable_gzip = false
    # https certs & key file
    cert_file =
    cert_key =
    # Unix socket path
    socket = /tmp/grafana.sock
    # CDN Url
    cdn_url =
  grafana2.txt: |
    ##################### Grafana Configuration Defaults #####################
    #
    # Do not modify this file in grafana installs
    #
    # possible values : production, development
    app_mode = production
    # instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
    instance_name = ${HOSTNAME}
    #################################### Paths ###############################
    [paths]
    # Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
    data = data
    # Temporary files in `data` directory older than given duration will be removed
    temp_data_lifetime = 24h
    # Directory where grafana can store logs
    logs = data/log
    # Directory where grafana will automatically scan and look for plugins
    plugins = data/plugins
    # folder that contains provisioning config files that grafana will apply on startup and while running.
    provisioning = conf/provisioning
    #################################### Server ##############################
    [server]
    # Protocol (http, https, h2, socket)
    protocol = http
    # The ip address to bind to, empty will bind to all interfaces
    http_addr =
    # The http port to use
    http_port = 3000
    # The public facing domain name used to access grafana from a browser
    domain = localhost
    # Redirect to correct domain if host header does not match domain
    # Prevents DNS rebinding attacks
    enforce_domain = false
    # The full public facing url
    root_url = %(protocol)s://%(domain)s:%(http_port)s/
    # Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
    serve_from_sub_path = false
    # Log web requests
    router_logging = false
    # the path relative working path
    static_root_path = public
    # enable gzip
    enable_gzip = false
    # https certs & key file
    cert_file =
    cert_key =
    # Unix socket path
    socket = /tmp/grafana.sock
    # CDN Url
    cdn_url =
    # Sets the maximum time in minutes before timing out read of an incoming request and closing idle connections.
kind: ConfigMap
metadata:
  name: grafana-config-gd6h8g92b6
  namespace: operations

kustomize 3.10.0 output

apiVersion: v1
data:
  grafana1.txt: |+
    ##################### Grafana Configuration Defaults #####################
    #
    # Do not modify this file in grafana installs
    #

    # possible values : production, development
    app_mode = production

    # instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
    instance_name = ${HOSTNAME}

    #################################### Paths ###############################
    [paths]
    # Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
    data = data

    # Temporary files in `data` directory older than given duration will be removed
    temp_data_lifetime = 24h

    # Directory where grafana can store logs
    logs = data/log

    # Directory where grafana will automatically scan and look for plugins
    plugins = data/plugins

    # folder that contains provisioning config files that grafana will apply on startup and while running.
    provisioning = conf/provisioning

    #################################### Server ##############################
    [server]
    # Protocol (http, https, h2, socket)
    protocol = http

    # The ip address to bind to, empty will bind to all interfaces
    http_addr =

    # The http port to use
    http_port = 3000

    # The public facing domain name used to access grafana from a browser
    domain = localhost

    # Redirect to correct domain if host header does not match domain
    # Prevents DNS rebinding attacks
    enforce_domain = false

    # The full public facing url
    root_url = %(protocol)s://%(domain)s:%(http_port)s/

    # Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
    serve_from_sub_path = false

    # Log web requests
    router_logging = false

    # the path relative working path
    static_root_path = public

    # enable gzip
    enable_gzip = false

    # https certs & key file
    cert_file =
    cert_key =

    # Unix socket path
    socket = /tmp/grafana.sock

    # CDN Url
    cdn_url =

  grafana2.txt: "##################### Grafana Configuration Defaults #####################\n#\n# Do not modify this file in grafana installs\n#\n\n# possible values : production, development\napp_mode = production\n\n# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty\ninstance_name = ${HOSTNAME}\n\n#################################### Paths ###############################\n[paths]\n# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)\ndata = data\n\n# Temporary files in `data` directory older than given duration will be removed\ntemp_data_lifetime = 24h\n\n# Directory where grafana can store logs\nlogs = data/log\n\n# Directory where grafana will automatically scan and look for plugins\nplugins = data/plugins\n\n# folder that contains provisioning config files that grafana will apply on startup and while running.\nprovisioning = conf/provisioning\n\n#################################### Server ##############################\n[server]\n# Protocol (http, https, h2, socket)\nprotocol = http\n\n# The ip address to bind to, empty will bind to all interfaces\nhttp_addr =\n\n# The http port to use\nhttp_port = 3000\n\n# The public facing domain name used to access grafana from a browser\ndomain = localhost\n\n# Redirect to correct domain if host header does not match domain\n# Prevents DNS rebinding attacks\nenforce_domain = false\n\n# The full public facing url\nroot_url = %(protocol)s://%(domain)s:%(http_port)s/\n\n# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.\nserve_from_sub_path = false\n\n# Log web requests\nrouter_logging = false\n\n# the path relative working path\nstatic_root_path = public\n\n# enable gzip\nenable_gzip = false\n\n# https certs & key file\ncert_file =\ncert_key =\n\n# Unix socket path\nsocket = /tmp/grafana.sock\n\n# CDN Url\ncdn_url =\n\n# Sets the maximum time in minutes before timing out read of an incoming request and closing idle connections. \n"
kind: ConfigMap
metadata:
  name: grafana-config-gf5dc5mb74
  namespace: operations

kustomize 4.1.2 output

apiVersion: v1
data:
  grafana1.txt: |+
    ##################### Grafana Configuration Defaults #####################
    #
    # Do not modify this file in grafana installs
    #

    # possible values : production, development
    app_mode = production

    # instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
    instance_name = ${HOSTNAME}

    #################################### Paths ###############################
    [paths]
    # Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
    data = data

    # Temporary files in `data` directory older than given duration will be removed
    temp_data_lifetime = 24h

    # Directory where grafana can store logs
    logs = data/log

    # Directory where grafana will automatically scan and look for plugins
    plugins = data/plugins

    # folder that contains provisioning config files that grafana will apply on startup and while running.
    provisioning = conf/provisioning

    #################################### Server ##############################
    [server]
    # Protocol (http, https, h2, socket)
    protocol = http

    # The ip address to bind to, empty will bind to all interfaces
    http_addr =

    # The http port to use
    http_port = 3000

    # The public facing domain name used to access grafana from a browser
    domain = localhost

    # Redirect to correct domain if host header does not match domain
    # Prevents DNS rebinding attacks
    enforce_domain = false

    # The full public facing url
    root_url = %(protocol)s://%(domain)s:%(http_port)s/

    # Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
    serve_from_sub_path = false

    # Log web requests
    router_logging = false

    # the path relative working path
    static_root_path = public

    # enable gzip
    enable_gzip = false

    # https certs & key file
    cert_file =
    cert_key =

    # Unix socket path
    socket = /tmp/grafana.sock

    # CDN Url
    cdn_url =

  grafana2.txt: "##################### Grafana Configuration Defaults #####################\n#\n#
    Do not modify this file in grafana installs\n#\n\n# possible values : production,
    development\napp_mode = production\n\n# instance name, defaults to HOSTNAME environment
    variable value or hostname if HOSTNAME var is empty\ninstance_name = ${HOSTNAME}\n\n####################################
    Paths ###############################\n[paths]\n# Path to where grafana can store
    temp files, sessions, and the sqlite3 db (if that is used)\ndata = data\n\n# Temporary
    files in `data` directory older than given duration will be removed\ntemp_data_lifetime
    = 24h\n\n# Directory where grafana can store logs\nlogs = data/log\n\n# Directory
    where grafana will automatically scan and look for plugins\nplugins = data/plugins\n\n#
    folder that contains provisioning config files that grafana will apply on startup
    and while running.\nprovisioning = conf/provisioning\n\n####################################
    Server ##############################\n[server]\n# Protocol (http, https, h2,
    socket)\nprotocol = http\n\n# The ip address to bind to, empty will bind to all
    interfaces\nhttp_addr =\n\n# The http port to use\nhttp_port = 3000\n\n# The public
    facing domain name used to access grafana from a browser\ndomain = localhost\n\n#
    Redirect to correct domain if host header does not match domain\n# Prevents DNS
    rebinding attacks\nenforce_domain = false\n\n# The full public facing url\nroot_url
    = %(protocol)s://%(domain)s:%(http_port)s/\n\n# Serve Grafana from subpath specified
    in `root_url` setting. By default it is set to `false` for compatibility reasons.\nserve_from_sub_path
    = false\n\n# Log web requests\nrouter_logging = false\n\n# the path relative working
    path\nstatic_root_path = public\n\n# enable gzip\nenable_gzip = false\n\n# https
    certs & key file\ncert_file =\ncert_key =\n\n# Unix socket path\nsocket = /tmp/grafana.sock\n\n#
    CDN Url\ncdn_url =\n\n# Sets the maximum time in minutes before timing out read
    of an incoming request and closing idle connections. \n"
kind: ConfigMap
metadata:
  name: grafana-config-gf5dc5mb74
  namespace: operations

Platform

macOS

[eddiezane]

/triage accepted
/assign @monopole

[ttk]

I also experienced this problem. Not sure what exactly triggers it the switch of the output, but in my testing, simply adding an additional if in the template was enough to do it.

I expect kustomize to always retain the same string syntax/format as the source. I've tested with various version from v.3.9.2 to the latest v.4.4.0 and they all have this problem.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[btheado]

The issue seems to surround overall length of files used in configmaps, but I have not been able to figure out exactly what is going on.

I encountered this issue today and I discovered one of the files I was loading via configMapGenerator contained a few lines with spaces at the end of the line. After I removed those spaces, the multi-line string changed back to the "easier to read" format.

I notice the last line of your grafana2.txt file contains a space at the end of the line:

"# Sets the maximum time in minutes before timing out read of an incoming request and closing idle connections. "

After I removed that space it also fixed the kustomize build output of your files.

Maybe configMapGenerator thinks it needs to use the quoted version of the string in order to preserve spaces at the end of lines?

If spaces at the end of lines are not important for your files, then the workaround is to just remove these spaces.

[btheado]

I discovered tab characters in the configMapGenerator loaded file will also trigger this behavior. Converting tabs to spaces works to restore the "easier to read" format.

[planeiii-te]

I've also seen multiline block conversions of - > end up as - | during the build as well. This has caused various issues.

meh.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: meh
spec:
  template:
    spec:
      containers:
        - image: alpine
          name: meh
          command:
            - /bin/sh
            - -c
            - >
              touch foo bar;

              mkdir catpants;
              cp foo \
                bar \
                catpants;

kustomize.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - meh.yaml

kustomize build . output:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: meh
spec:
  template:
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - |
          touch foo bar;
          mkdir catpants; cp foo \

            bar \
            catpants;
        image: alpine
        name: meh

Notice how the - > which converts new lines to spaces is replaced with - | which maintains new lines. This causes the call to cp to fail with the (randomly inserted) blank line.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ttreptow]

Encountered a file that had emoji characters that also triggered the behavior

[casper-onboard]

I have the same issue, using a ConfigMapGenerator with around 30 files, and the generated ConfigMap switches around the order of some pieces of text every time (the text in some files is identical, which may contribute to the issue).

This causes kubectl diff to show changes that aren't really changes. Any workaround to this?