podman inspect filters not working in podman-2.2.1 to podman-3.2.0_rc1

[srcshelton]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman inspect -f '{{ <filter> }}' <image> with certain filter strings no longer works with podman-2.2.1.

Describe the results you received:

The command:

podman inspect --format "imageId: {{.Id}} size: {{.Size}}" <image>

... (which comes from the podman help output) works as expected. However:

podman inspect --format '{{ .Config.Env }}' <image>

... returns no output.

podman inspect <image> | jq '.[].Config.Env'

... however, still does (demonstrating that the data is there)

Describe the results you expected:

As previous releases, podman inspect -f '{{ .Config.Env }}' <image> would display the environment variables embedded within the specified image.

Output of podman version:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Git Commit:   a0d478edea7f775b7ce32f8eb1a01e75374486cb
Built:        Wed Dec  9 11:33:11 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 35a2fa83022e56e18af7e6a865ba5d7165fa2a4a'
  cpus: 8
  distribution:
    distribution: gentoo
    version: unknown
  eventLogger: file
  hostname: dellr330
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.9.6-gentoo
  linkmode: dynamic
  memFree: 871362560
  memTotal: 33392836608
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 21837553664
  swapTotal: 25769787392
  uptime: 615h 32m 37.93s (Approximately 25.62 days)
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: localhost:5000
  search:
  - docker.io
  - docker.pkg.github.com
  - quay.io
  - public.ecr.aws
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 26
    paused: 0
    running: 23
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "false"
  graphRoot: /space/podman/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 466
  runRoot: /space/podman/run
  volumePath: /space/podman/volumes
version:
  APIVersion: 2.1.0
  Built: 1607513591
  BuiltTime: Wed Dec  9 11:33:11 2020
  GitCommit: a0d478edea7f775b7ce32f8eb1a01e75374486cb
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

[srcshelton]

As a work-around, the following works:

buildah inspect --format '{{ .OCIv1.Config.Env }}' <image>

[mheon]

@jwhonce PTAL

[Luap99]

It works for me with 2.2.1

$ podman inspect --format '{{ .Config.Env }}' alpine
[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin]
$ podman version 
Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 15:37:50 2020
OS/Arch:      linux/amd64

[srcshelton]

The alpine example above works for me too - so it doesn't appear to be affecting all images, but it's certainly affecting some.
And, as noted, the data is there to be extracted, as filtering through jq can get the same output.
But there's definitely a regression here, as earlier releases did not have this issue (with images built in exactly the same way).

[Luap99]

Can you share a reproducer? It is working with all my images.

[srcshelton]

Yes, I do!

If you build this Dockerfile https://raw.githubusercontent.com/srcshelton/docker-gentoo-build/master/gentoo-base/Dockerfile.env, which exists solely of ENV declarations, then sudo podman inspect -f '{{ .Config.Env }}' gentoo-env:latest returns nothing whilst sudo buildah inspect --format '{{ .OCIv1.Config.Env }}' gentoo-env:latest outputs the expected data.

Prior releases of podman did not have this issue.

[Luap99]

I can reproduce with your file.

git bisect says 3cca814 @jwhonce PTAL

A simpler reproducer would be using this Containerfile:

FROM alpine

ENV TEST="	t"

Notice that this is a tab character before the t.

The problem is caused by the use of the tabwriter:

podman/cmd/podman/inspect/inspect.go

Lines 261 to 262 in 142b4ac

	w := tabwriter.NewWriter(os.Stdout, 8, 2, 2, ' ', 0)
	return t.Execute(w, data)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[baude]

@jwhonce ptal

[rhatdan]

A little late for podman 2.1.1 with Podman 3.0 about to release can we just close this issue?

[srcshelton]

# podman version
Version:      3.0.0-rc2
API Version:  3.0.0
Go Version:   go1.15.7
Git Commit:   510a983a752a136a5df0bdbff9d14db0490956b2
Built:        Fri Feb  5 20:18:56 2021
OS/Arch:      linux/amd64
# podman inspect --format '{{ .Config.Env }}' gentoo-env
# podman inspect gentoo-env | jq '.[].Config.Env'
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  ...
]

... it's not fixed in RC2 either :(

[mheon]

@jwhonce assures us that it will be ready for 3.0 final - we apparently need a revendor of containers/common with his patches, and then a fresh patch to Podman on top of that.

[rhatdan]

Looks like this is working in the main branch now.

podman inspect --format '{{ .Config.Env }}' -l
[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm container=oci HOME=/ HOSTNAME=c571ef8e3531]

[Luap99]

This is not fixed. @rhatdan You need to have a tab character in the env var. Use the Containerfile from this comment #8785 (comment)

[rhatdan]

Ok, I looked at the first commands and tried them locally and had no issue. I did not read down the the reproducer.

[srcshelton]

Still an issue in podman-3.1.2 and podman-3.2.0_rc1 - please could we try to resolve this before podman-3.2.0 is released?

[vrothberg]

@jwhonce PTAL

[jwhonce]

@vrothberg Looked, use of []interface{} in the inspect code makes a quick fix difficult.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 do you have time to tackle this?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@jwhonce @Luap99 @srcshelton Is this still an issue?

[srcshelton]

Still broken with podman-3.2.3, yes :(

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc PTAL

[flouthoc]

@srcshelton I think problem is we are never flushing the writer, this should fix things for you Please try with #11499 . I will backport once you confirm.

[flouthoc]

@Luap99 This should also fix the reproducer (#8785 (comment)) which you provided. Could you please give this commit a shot.

[srcshelton]

Just to say, this all appears fixed with 3.4.1 - thanks!

[flouthoc]

@srcshelton 👍 thanks for confirming.