cacheFrom option doesn't work

[zhorzh]

Thanks a lot for building skaffold!
I feel like I will not need any CI/CD tools soon :) ...almost.

I want to speed up the container building process for my project on Google Cloud.
The problem is that cacheFrom option is ignored completely.

Expected behavior

cacheFrom should force build process to use cache from the previously built image

Actual behavior

build process repeats every time without using the cache

Information

I have checked the previously build image on gcr.io - it exists

• Skaffold version: v0.12.0
• Operating system: mac os 10.13.3
• Contents of skaffold.yaml:

  - name: stage
    build:
      googleCloudBuild:
        projectId: neural-pub
      artifacts:
        - imageName: gcr.io/neural-pub/api
          workspace: ./api
          docker:
            cacheFrom:
            - gcr.io/neural-pub/api
        - imageName: gcr.io/neural-pub/web
          workspace: ./web
          docker:
            cacheFrom:
            - gcr.io/neural-pub/web
    deploy:
      kubectl:
        manifests:
          - ./ingress/k8s-stage-*
          - ./api/k8s-stage-*
          - ./web/k8s-stage-*

Steps to reproduce the behavior

1. skaffold run --profile stage
2. run this command again

I am wondering - does this feature work on Google Cloud?

[dgageot]

Thanks @zhorzh for reporting this issue. Indeed, a fix is needed to make this work on Cloud Builder. the images used in cacheFrom should be pulled before hand. See https://cloud.google.com/cloud-build/docs/speeding-up-builds#using_a_cached_docker_image

[radityasurya]

the cache-from also not working on circle ci, but it is working locally

[balopat]

I'm thinking that skaffold could help with this - it would be super convenient if the docker cacheFrom would pull the image - given the state of CI systems. I just ran into this myself, and it's annoying.
Skaffold could check the existence of the cacheFrom images in the local docker daemon and if it's not there pull it.

[dgageot]

big +1 on that

[demisx]

Can some one please explain how skaffold dev --cache-artifacts is different from cacheFrom: used in skaffold YAML files? Maybe I can help to improve the docs. I've read them few times, but still don't get when to use one vs. the other:

# From skaffold dev -h
--cache-artifacts  -- Set to true to enable caching of artifacts

# On https://skaffold.dev/docs/references/yaml/
cacheFrom:  -- The Docker images used as cache sources.

[priyawadhwa]

Hi @demisx

The --cache-artifacts flags is for caching skaffold artifacts. Skaffold stores its own cache in ~/.skaffold/cache, and allows skaffold to skip rebuilding artifacts it has built before.

cacheFrom is a docker CLI flag specifically for artifacts built by docker, and is used by docker to determine images to consider as cache sources when building a new docker image.

[demisx]

Hi @priyawadhwa. Thank you so much for your explanation. Would you say these would be the best practices for a common dev environment in order to speed up the experience?

1. Pass --cache-artifacts --no-prune to skaffold dev on each run
2. In skaffold.yaml reference the built image in cacheFrom

build:
    artifacts:    
      - image: dl-org-db
          docker:
            dockerfile: path/to/Dockerfile
            cacheFrom:
              - dl-org-db # <- same image name as in "image" field?

[priyawadhwa]

1. I would definitely recommend --cache-artifacts to speed up the dev loop. Whether or not --no-prune is a good option really depends on each use case, and whether a user can afford to store images locally.

2. You could do that if there's a chance the image has been built for, but really any images that may share layers with the artifact you're trying to build would make sense here.

[demisx]

If I remove --no-prune option, won't that force skaffold to build images on each run? I have a feeling that I am misunderstanding what kind of artifacts skaffold builds and what skaffold relies on docker daemon to build.

[priyawadhwa]

Ah sorry I didn't clarify, images built for the artifact cache aren't deleted locally during prune!