WARN[0001] image [ccoehp.azurecr.io/database-client/postgres-psql] is not used by the deployment + ACR + AKS

[anikm1987]

Expected behavior

Warning should not come and Skaffold should use the image properly. Not sure if it is causing the issue while doing sync

Actual behavior

Information

• Skaffold version: version...
  6b8b50a - Was running from HEAD version for the nil pointer issue fix
• Operating system: ...
  Linux
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta5
kind: Config
metadata:
  name: postgres-aks-test
build:
  tagPolicy:
     sha256: {}
  artifacts:
  - image: ccoehp.azurecr.io/database-client/postgres-psql
    context: ububtu-postgres-client
deploy:
  kubectl:
    manifests:
    - ububtu-postgres-client/manifest/pg_app_deployment.yaml
   

Steps to reproduce the behavior

Every deployment I am running I am getting the same warning.

1. a clonable repository with the sample skaffold project
2. skaffold dev --port-forward -vdebug -n ccoe-hp-devtest
3. Deployment yaml config

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-test
  template:
    metadata:
      labels:
        app: postgres-test
    spec:
      containers:
        - name: postgres-test
          image: ccoehp.azurecr.io/database-client/postgres-psql
      imagePullSecrets:
        - name: ccoehpacrsecret

4. After deployment I crosschecked by describing the pod, it is showing correct image name with hashed value

Containers:
  postgres-test:
    Container ID:   docker://5ccdddd9893d721be53a836a5c0d90de8856aef7f82b2efdae6eb0d7feda6c1a
    Image:          ccoehp.azurecr.io/database-client/postgres-psql
    Image ID:       docker-pullable://ccoehp.azurecr.io/database-client/postgres-psql@sha256:78ec26e82cc69d70a4a8f84fb6a0b807beeb6b6e33d64d88fae6b0aaf7d8e565

5. Full log file in debug mode

skaffold dev --port-forward -vdebug -n ccoe-hp-devtest
INFO[0000] starting gRPC server on port 50051           
INFO[0000] starting gRPC HTTP server on port 50052      
INFO[0000] Skaffold &{Version:6b8b50a35 ConfigVersion:skaffold/v2beta5 GitVersion: GitCommit:6b8b50a35f3436878ceef682193f1a2b05cab6bf GitTreeState:clean BuildDate:2020-07-05T14:15:23Z GoVersion:go1.14.4 Compiler:gc Platform:linux/amd64} 
DEBU[0000] found config for context "akscluster00001"   
DEBU[0000] Defaulting build type to local build         
DEBU[0000] validating yamltags of struct SkaffoldConfig 
DEBU[0000] validating yamltags of struct Metadata       
DEBU[0000] validating yamltags of struct Pipeline       
DEBU[0000] validating yamltags of struct BuildConfig    
DEBU[0000] validating yamltags of struct Artifact       
DEBU[0000] validating yamltags of struct ArtifactType   
DEBU[0000] validating yamltags of struct DockerArtifact 
DEBU[0000] validating yamltags of struct TagPolicy      
DEBU[0000] validating yamltags of struct ShaTagger      
DEBU[0000] validating yamltags of struct BuildType      
DEBU[0000] validating yamltags of struct LocalBuild     
DEBU[0000] validating yamltags of struct DeployConfig   
DEBU[0000] validating yamltags of struct DeployType     
DEBU[0000] validating yamltags of struct KubectlDeploy  
DEBU[0000] validating yamltags of struct KubectlFlags   
INFO[0000] Using kubectl context: akscluster00001       
DEBU[0000] Using builder: local                         
DEBU[0000] setting Docker user agent to skaffold-6b8b50a35 
DEBU[0000] push value not present, defaulting to true because localCluster is false 
Listing files to watch...
 - ccoehp.azurecr.io/database-client/postgres-psql
DEBU[0000] Found dependencies for dockerfile: []        
INFO[0000] List generated in 1.3024ms                   
Generating tags...
 - ccoehp.azurecr.io/database-client/postgres-psql -> ccoehp.azurecr.io/database-client/postgres-psql:latest
INFO[0000] Tags generated in 72.2µs                     
Checking cache...
DEBU[0000] Found dependencies for dockerfile: []        
INFO[0000] update check failed: get latest and current Skaffold version: parsing current semver, skipping update check: parsing semver: No Major.Minor.Patch elements found 
DEBU[0000] could not parse data                         
 - ccoehp.azurecr.io/database-client/postgres-psql: Found Remotely
INFO[0001] Cache check complete in 988.6814ms           
Tags used in deployment:
 - ccoehp.azurecr.io/database-client/postgres-psql -> ccoehp.azurecr.io/database-client/postgres-psql:latest@sha256:78ec26e82cc69d70a4a8f84fb6a0b807beeb6b6e33d64d88fae6b0aaf7d8e565
DEBU[0001] getting client config for kubeContext: ``    
Starting deploy...
DEBU[0001] Running command: [kubectl version --client -ojson] 
DEBU[0001] Command output: [{
  "clientVersion": {
    "major": "1",
    "minor": "18",
    "gitVersion": "v1.18.5",
    "gitCommit": "e6503f8d8f769ace2f338794c914a96fc335df0f",
    "gitTreeState": "clean",
    "buildDate": "2020-06-26T03:47:41Z",
    "goVersion": "go1.13.9",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}
] 
DEBU[0001] Running command: [kubectl --context akscluster00001 --namespace ccoe-hp-devtest create --dry-run=client -oyaml -f /datadrive/kubernetes/postgres-aks-test/ububtu-postgres-client/manifest/pg_app_deployment.yaml] 
DEBU[0001] Command output: [apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres-test
  namespace: ccoe-hp-devtest
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-test
  template:
    metadata:
      labels:
        app: postgres-test
    spec:
      containers:
      - image: ccoehp.azurecr.io/database-client/postgres-psql
        name: postgres-test
      imagePullSecrets:
      - name: ccoehpacrsecret
] 
WARN[0001] image [ccoehp.azurecr.io/database-client/postgres-psql] is not used by the deployment 
DEBU[0001] manifests with tagged images: apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres-test
  namespace: ccoe-hp-devtest
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-test
  template:
    metadata:
      labels:
        app: postgres-test
    spec:
      containers:
      - image: ccoehp.azurecr.io/database-client/postgres-psql
        name: postgres-test
      imagePullSecrets:
      - name: ccoehpacrsecret 
DEBU[0002] manifests with labels apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/managed-by: skaffold-6b8b50a35
    skaffold.dev/builder: local
    skaffold.dev/cleanup: "true"
    skaffold.dev/deployer: kubectl
    skaffold.dev/docker-api-version: "1.40"
    skaffold.dev/namespace: ccoe-hp-devtest
    skaffold.dev/run-id: 1c7b6b36-bc03-4ab5-a66d-bac6916dd626
    skaffold.dev/tag-policy: sha256
    skaffold.dev/tail: "true"
  name: postgres-test
  namespace: ccoe-hp-devtest
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-test
  template:
    metadata:
      labels:
        app: postgres-test
    spec:
      containers:
      - image: ccoehp.azurecr.io/database-client/postgres-psql
        name: postgres-test
      imagePullSecrets:
      - name: ccoehpacrsecret 
DEBU[0002] 1 manifests to deploy. 1 are updated or new  
DEBU[0002] Running command: [kubectl --context akscluster00001 --namespace ccoe-hp-devtest apply -f - --force --grace-period=0] 
 - deployment.extensions/postgres-test created
INFO[0002] Deploy complete in 1.2503068s                
Waiting for deployments to stabilize...
DEBU[0002] getting client config for kubeContext: ``    
DEBU[0002] checking status ccoe-hp-devtest:deployment/postgres-test 
DEBU[0002] Running command: [kubectl --context akscluster00001 --namespace ccoe-hp-devtest rollout status deployment postgres-test --namespace ccoe-hp-devtest --watch=false] 
DEBU[0003] Command output: [Waiting for deployment "postgres-test" rollout to finish: 0 of 1 updated replicas are available...
] 
DEBU[0003] Running command: [kubectl --context akscluster00001 --namespace ccoe-hp-devtest rollout status deployment postgres-test --namespace ccoe-hp-devtest --watch=false] 
 - ccoe-hp-devtest:deployment/postgres-test: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0004] Command output: [Waiting for deployment "postgres-test" rollout to finish: 0 of 1 updated replicas are available...
] 
DEBU[0004] Running command: [kubectl --context akscluster00001 --namespace ccoe-hp-devtest rollout status deployment postgres-test --namespace ccoe-hp-devtest --watch=false] 
DEBU[0004] Command output: [Waiting for deployment "postgres-test" rollout to finish: 0 of 1 updated replicas are available...
] 
DEBU[0004] Running command: [kubectl --context akscluster00001 --namespace ccoe-hp-devtest rollout status deployment postgres-test --namespace ccoe-hp-devtest --watch=false] 
DEBU[0005] Command output: [Waiting for deployment "postgres-test" rollout to finish: 0 of 1 updated replicas are available...
] 
DEBU[0005] Running command: [kubectl --context akscluster00001 --namespace ccoe-hp-devtest rollout status deployment postgres-test --namespace ccoe-hp-devtest --watch=false] 
DEBU[0006] Command output: [Waiting for deployment "postgres-test" rollout to finish: 0 of 1 updated replicas are available...
] 
DEBU[0006] Running command: [kubectl --context akscluster00001 --namespace ccoe-hp-devtest rollout status deployment postgres-test --namespace ccoe-hp-devtest --watch=false] 
DEBU[0006] Command output: [Waiting for deployment "postgres-test" rollout to finish: 0 of 1 updated replicas are available...
] 
DEBU[0007] Running command: [kubectl --context akscluster00001 --namespace ccoe-hp-devtest rollout status deployment postgres-test --namespace ccoe-hp-devtest --watch=false] 
DEBU[0007] Command output: [deployment "postgres-test" successfully rolled out
] 
 - ccoe-hp-devtest:deployment/postgres-test is ready.
Deployments stabilized in 5.1982239s
DEBU[0007] getting client config for kubeContext: ``    
DEBU[0007] getting client config for kubeContext: ``    
Press Ctrl+C to exit
DEBU[0007] Change detected <nil>                        
DEBU[0007] Found dependencies for dockerfile: []        

[anikm1987]

I doubt this is one of the reason why live sync is not working for me.
Getting below warning as well for another POD
Syncing 1 files for ccoehp.azurecr.io/hp-db-app:latest@sha256:189aee6e6e32ecde2c4ef28d271c964727b2751c7fcf5b075d210bc811753b38
INFO[1198] Copying files: map[app/models/azc.model.js:[/usr/src/app/app/models/azc.model.js]] to ccoehp.azurecr.io/hp-db-app:latest@sha256:189aee6e6e32ecde2c4ef28d271c964727b2751c7fcf5b075d210bc811753b38
DEBU[1198] getting client config for kubeContext: ``
WARN[1199] Skipping deploy due to sync error: copying files: didn't sync any files
Watching for changes...

[briandealwis]

The problem is that your deployment version is extensions/v1beta1 which has long been deprecated (current version is apps/v1), and wasn't supported by Skaffold until just recently (#4478, not yet in a release).

Skaffold normally processes the configured Kubernetes resource manifests to rewrite images to reference the built versions. Skaffold v1.8.0 introduced a feature to limit rewriting to an allowed list of resource types (#3833). extensions/v1beta1 was not included.

Switching your deployment to apps/v1 should allow it to work.

[briandealwis]

Closing as I believe we have identified the root cause and there is a fix in the next release.