ACR build fails with: failed to download context. Please check if the URL is incorrect.

[ezYakaEagle442]

See Runner logs at https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3191241482/jobs/5207516396

Workflow at https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/blob/main/.github/workflows/maven-build.yml#L167

env:
  AZURE_CONTAINER_REGISTRY: acrpetcliaca 
  REGISTRY_URL: acrpetcliaca.azurecr.io  # set this to the URL of your registry
  REPOSITORY: petclinic                  # set this to your ACR repository
  PROJECT_NAME: petclinic                # set this to your project's name
  KV_NAME: kv-petcliaca442               
  
  RG_KV: rg-iac-kv777 
  RG_APP: rg-iac-aca-petclinic-mic-srv 

  # ==== APPS ====
  PRJ_PREFIX: aca-spring-petclinic

  API_GATEWAY: api-gateway
  ADMIN_SERVER: admin-server
  CUSTOMERS_SERVICE: customers-service
  VETS_SERVICE: vets-service
  VISITS_SERVICE: visits-service

  CONFIG_SERVER: config-server
  DISCOVERY_SERVER: discovery-server

  # https://github.com/Azure/actions-workflow-samples/blob/master/assets/create-secrets-for-GitHub-workflows.md#consume-secrets-in-your-workflow
  # https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-using-secrets
  # ==== Secrets ====
  credentials: ${{ secrets.AZURE_CREDENTIALS }}
  AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
  SPN_ID: ${{ secrets.SPN_ID }}
  SPN_PWD: ${{ secrets.SPN_PWD }}
  
    - name: admin-server ACR build
      id: admin-server-acr-build
      uses: azure/acr-build@v1
      with:
        service_principal: ${{ env.SPN_ID }} # The Service Principal credentials : must be stored in KV
        service_principal_password: ${{ env.SPN_PWD }} # The Service Principal credentials: must be stored in KV
        tenant: ${{ env.AZURE_TENANT_ID }} # The ACR tenant
        registry: ${{ env.AZURE_CONTAINER_REGISTRY }} # The ACR name
        repository: ${{ env.REPOSITORY }} # The repository on the ACR
        image: ${{ env.ADMIN_SERVER }} # Docker image name. Default tag: The action will default to use the first 8 characters of the commit SHA if no tag is specified. Tagged with GitHub commit ID (SHA), ex: 'mcr.microsoft.com/azuredocs/containerapps-helloworld:latest'
        # git_access_token: ${{ secrets.git_access_token }} # The Github access token for private repositories
        folder: ${{ github.workspace }} # Build context for Docker agent
        dockerfile: ./docker/petclinic-admin-server/Dockerfile # Path to the Dockerfile relative to `folder`
        branch: main # Branch to root the context off of

Building Docker image petclinic/admin-server:39c27f0e from ezYakaEagle442/aca-java-petclinic-mic-srv on main and using context /github/workspace ; and pushing it to acrpetcliaca Azure Container Registry
Logging into azure..
[
  ***
    "cloudName": "AzureCloud",
    "homeTenantId": "***",
    "id": "7b5f97dc-3c4d-424d-[82](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3191241482/jobs/5207516396#step:10:83)88-bdde3891f242",
    "isDefault": true,
    "managedByTenants": [],
    "name": "XXX Internal subscription",
    "state": "Enabled",
    ***
    "user": ***
      "name": "***",
      "type": "servicePrincipal"
    ***
  ***
]
Sending build job to ACR..
WARNING: Sending context to registry: acrpetcliaca...
WARNING: Queued a build with ID: cb6
WARNING: Waiting for an agent...
2022/10/05 17:21:09 Downloading source code...
Run ID: cb6 failed after 2s. Error: failed to download context. Please check if the URL is incorrect. If it has credentials, please check if they are expired
ERROR: Run failed

see also Azure/azure-cli#20761

@ams0

[ezYakaEagle442]

The Dockerfile is at https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/blob/main/docker/petclinic-admin-server/Dockerfile

I can locally successfully build the image :

docker build --build-arg --no-cache -t "petclinic-admin-server" -f "./docker/petclinic-admin-server/Dockerfile" .
docker run -p 8080:8080 -p 8081:8081 -p 8888:8888 -p 9090:9090 petclinic-admin-server 

I do not have the same error when using CLI 2.40.0 in the workflow :

          az acr build --registry ${{ env.AZURE_CONTAINER_REGISTRY }} -g ${{ env.RG_APP }} \
            -t ${{ env.REPOSITORY }}/${{ env.ADMIN_SERVER }}:{{.Run.ID}} \
            --file "./docker/petclinic-admin-server/Dockerfile" \
            ${{ github.workspace }}

Run acr_usr=$(az deployment group show -g rg-iac-aca-petclinic-mic-srv -n acrpetcliaca --query properties.outputs.acrRegistryUsr.value | tr -d '"')
WARNING: Packing source code into tar to upload...
WARNING: Excluding '.gitignore' based on default ignore rules
WARNING: Excluding '.git' based on default ignore rules
WARNING: Uploading archived source code from '/tmp/build_archive_4f5ebbdb5430456890cb403d23118da8.tar.gz'...
WARNING: Sending context (362.578 MiB) to registry: acrpetcliaca...
WARNING: Queued a build with ID: cb7
WARNING: Waiting for an agent...
2022/10/05 21:06:41 Downloading source code...
2022/10/05 21:06:47 Finished downloading source code
2022/10/05 21:06:48 Using acb_vol_75c4b778-fa03-4499-9288-cf2e8a3c5416 as the home volume
2022/10/05 21:06:48 Setting up Docker configuration...
2022/10/05 21:06:49 Successfully set up Docker configuration
2022/10/05 21:06:49 Logging in to registry: acrpetcliaca.azurecr.io
2022/10/05 21:06:50 Successfully logged into acrpetcliaca.azurecr.io
2022/10/05 21:06:50 Executing step ID: build. Timeout(sec): 28800, Working directory: '', Network: ''
2022/10/05 21:06:50 Scanning for dependencies...
2022/10/05 21:06:50 Successfully scanned dependencies
2022/10/05 21:06:50 Launching container with name: build
Sending build context to Docker daemon    435MB

Step 1/24 : FROM mcr.microsoft.com/openjdk/jdk:11-mariner as builder
11-mariner: Pulling from openjdk/jdk
7f0517a4d0b4: Pulling fs layer
89d1ee0edfd7: Pulling fs layer
7f0517a4d0b4: Verifying Checksum
7f0517a4d0b4: Download complete
7f0517a4d0b4: Pull complete
89d1ee0edfd7: Verifying Checksum
89d1ee0edfd7: Download complete
89d1ee0edfd7: Pull complete
Digest: sha256:e01f4e2b5cdca94b0e2b6c446ac8ef811823d82380284bd88e73a4c33df0b8c9
Status: Downloaded newer image for mcr.microsoft.com/openjdk/jdk:11-mariner
 ---> 3e383202dc94
Step 2/24 : ENV APPI_VERSION="3.4.1"
 ---> Running in 8591c58b01e8
Removing intermediate container 8591c58b01e8
 ---> 81d72436f7db
Step 3/24 : ENV APP_INSIGHTS_AGENT_JAR_FILE_PATH="/tmp/app/applicationinsights-agent-$***APPI_VERSION***.jar"
 ---> Running in 139969f5b37b
Removing intermediate container 139969f5b37b
 ---> 9d346709ee21
Step 4/24 : LABEL Maintainer="pinpin <[email protected]>"
 ---> Running in 4e79024d8fda
Removing intermediate container 4e79024d8fda
 ---> 95800f62de61
Step 5/24 : LABEL Description="PetClinic admin-server: Java Spring Boot microservice built from MS OpenJDK 11-Mariner"
 ---> Running in 2ce1a3d8e0e7
Removing intermediate container 2ce1a3d8e0e7
 ---> 517d2febd55e
Step 6/24 : RUN java -version
 ---> Running in 7bd2803dc43f
openjdk version "11.0.16.1" 2022-08-12 LTS
OpenJDK Runtime Environment Microsoft-40648 (build 11.0.16.1+1-LTS)
OpenJDK 64-Bit Server VM Microsoft-40648 (build 11.0.16.1+1-LTS, mixed mode, sharing)
Removing intermediate container 7bd2803dc43f
 ---> d3adeb20a8e3
Step 7/24 : RUN ls -al /usr/lib/jvm/msopenjdk-11
 ---> Running in 28d12fb3e87d
total 40
drwxr-xr-x  9 root root 4096 Oct  3 09:04 .
drwxr-xr-x  3 root root 4096 Oct  3 09:04 ..
drwxr-xr-x  2 root root 4096 Oct  3 09:04 bin
drwxr-xr-x  4 root root 4096 Oct  3 09:04 conf
drwxr-xr-x  3 root root 4096 Oct  3 09:04 include
drwxr-xr-x  2 root root 4096 Oct  3 09:04 jmods
drwxr-xr-x 73 root root 4096 Aug 24 21:23 legal
drwxr-xr-x  6 root root 4096 Oct  3 09:04 lib
drwxr-xr-x  4 root root 4096 Aug 24 21:23 man
-rw-rw-r--  1 root root 1232 Aug 24 21:23 release
Removing intermediate container 28d12fb3e87d
 ---> 2f150b4d5535
Step 8/24 : RUN mkdir /tmp/app
 ---> Running in 8fd3f7061[98](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:99)2
Removing intermediate container 8fd3f7061982
 ---> 917812ab34c4
Step 9/24 : WORKDIR /tmp/app
 ---> Running in 1cf32a6705d4
Removing intermediate container 1cf32a6705d4
 ---> 29427d6cad0b
Step 10/24 : COPY "spring-petclinic-admin-server/target/*.jar" /tmp/app/app.jar
 ---> 82e1eb83509a
Step 11/24 : RUN curl -SL --output $***APP_INSIGHTS_AGENT_JAR_FILE_PATH*** https://github.com/microsoft/ApplicationInsights-Java/releases/download/$***APPI_VERSION***/applicationinsights-agent-$***APPI_VERSION***.jar
 ---> Running in 6f9d1fb7dae4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  3 36.4M    3 1135k    0     0  1072k      0  0:00:34  0:00:01  0:00:33 1072k
 40 36.4M   40 14.8M    0     0  7401k      0  0:00:05  0:00:02  0:00:03 13.8M
 84 36.4M   84 30.8M    0     0  10.0M      0  0:00:03  0:00:03 --:--:-- 14.8M
100 36.4M  100 36.4M    0     0  10.7M      0  0:00:03  0:00:03 --:--:-- 15.1M
Removing intermediate container 6f9d1fb7dae4
 ---> e46dd8572f08
Step 12/24 : RUN java -Djarmode=layertools -jar "/tmp/app/app.jar" extract
 ---> Running in 5e7847a82b8f
Removing intermediate container 5e7847a82b8f
 ---> 4b758dfb2c5d
Step 13/24 : FROM mcr.microsoft.com/openjdk/jdk:11-mariner
11-mariner: Pulling from openjdk/jdk
Digest: sha256:e01f4e2b5cdca94b0e2b6c446ac8ef811823d82380284bd88e73a4c33df0b8c9
Status: Image is up to date for mcr.microsoft.com/openjdk/jdk:11-mariner
 ---> 3e383202dc94
Step 14/24 : WORKDIR /tmp/app
 ---> Running in 1d3[99](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:100)750b0a0
Removing intermediate container 1d399750b0a0
 ---> b8243eff63b3
Step 15/24 : ENV APPI_VERSION="3.4.1"
 ---> Running in 6ee529833133
Removing intermediate container 6ee529833133
 ---> 788d2c0e8dae
Step 16/24 : ENV APP_INSIGHTS_AGENT_JAR_FILE_PATH="/tmp/app/applicationinsights-agent-$***APPI_VERSION***.jar"
 ---> Running in c787d2eab2ac
Removing intermediate container c787d2eab2ac
 ---> da6131d215d8
Step 17/24 : ENV SPRING_PROFILES_ACTIVE="docker,mysql"
 ---> Running in b827f9aa80ad
Removing intermediate container b827f9aa80ad
 ---> a5f95f70f3ba
Step 18/24 : COPY --from=builder $***APP_INSIGHTS_AGENT_JAR_FILE_PATH*** ./
 ---> d225cc4f8866
Step 19/24 : COPY --from=builder /tmp/app/dependencies/ ./
 ---> be3050560a49
Step 20/24 : COPY --from=builder /tmp/app/snapshot-dependencies/ ./
 ---> 64de4376ec11
Step 21/24 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
failed to export image: failed to create image: failed to get layer sha256:7f92e557[119](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:120)03c43c92210c264f20871f1ad724fc4373ed2383be03e908e[121](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:122)0: layer does not exist
[202](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:203)2/10/05 21:07:27 Container failed during run: build. No retries remaining.
failed to run step ID: build: exit status 1

Run ID: cb7 failed after 47s. Error: failed during run, err: exit status 1
ERROR: Run failed
Error: Process completed with exit code 1.

[AntonioVT]

Hi @ezYakaEagle442 ,

Looking into the logs, it seems that there's an issue at Step 21/24. It looks like steps 18/19/20 succeeded when copying from the expected directories, but for some reason the directory /tmp/app/spring-boot-loader/ fails.

1. Which step/process is responsible of creating the directory tmp/app/sprint-boot-loader?
2. I know you mentioned the build is working fine locally. Could you verify the following, please?
   2.1 Could you make sure there's no existing directory tmp/app/sprint-boot-loader in your system?
   2.2 Could you make sure the other directories (/tmp/app/dependencies/, tmp/app/snapshot-dependencies/, etc) are created as expected?

[ezYakaEagle442]

@AntonioVT thank you for your quick reply.

1. The step is

RUN java -Djarmode=layertools -jar "/tmp/app/app.jar" extract

2. I do not think there is anythingthere as the final stage copies the artifacts to the folder /tmp/app :

COPY --from=builder ${APP_INSIGHTS_AGENT_JAR_FILE_PATH} ./
COPY --from=builder /tmp/app/dependencies/ ./
COPY --from=builder /tmp/app/snapshot-dependencies/ ./
COPY --from=builder /tmp/app/spring-boot-loader/ ./
COPY --from=builder /tmp/app/application/ ./

I als odid a quick test locally in VS, ACR CLI fails with the same error

az acr build --registry acrpetcliaca -g rg-iac-aca-petclinic-mic-srv -t petclinic:adm-test --file "./docker/petclinic-admin-server/Dockerfile" .

Step 21/24 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
failed to export image: failed to create image: failed to get layer sha256:c37bbfdfd933d6d6b3fafc0df403c8a79824968a020966b2821da6f358a5df60: layer does not exist 
2022/10/06 09:11:38 Container failed during run: build. No retries remaining.
failed to run step ID: build: exit status 1

Run ID: cb8 failed after 45s. Error: failed during run, err: exit status 1

[ezYakaEagle442]

confirmed , I have added a debug trace with :

RUN ls -al /tmp/app

the folder is empty:

---> Running in 28f2333e7851
Removing intermediate container 28f2333e7851
 ---> 99125d2d9ad3
Step 18/25 : RUN ls -al /tmp/app
 ---> Running in 30ec769fe2c4
total 8
drwxr-xr-x 2 root root 4096 Oct  6 09:22 .
drwxrwxrwt 1 root root 4096 Oct  6 09:22 ..
Removing intermediate container 30ec769fe2c4
 ---> 21078588665e
Step 19/25 : COPY --from=builder ${APP_INSIGHTS_AGENT_JAR_FILE_PATH} ./
 ---> 4f49097c12d6
Step 20/25 : COPY --from=builder /tmp/app/dependencies/ ./
 ---> 26220f3c83e0
Step 21/25 : COPY --from=builder /tmp/app/snapshot-dependencies/ ./
 ---> 6da4249d8e60
Step 22/25 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
failed to export image: failed to create image: failed to get layer sha256:498b96bfc7390a186bae3f682265d00b334865a11d75e42eb7a4449aa6e2e6cd: layer does not exist 
2022/10/06 09:22:32 Container failed during run: build. No retries remaining.
failed to run step ID: build: exit status 1

Run ID: cb9 failed after 43s. Error: failed during run, err: exit status 1

[AntonioVT]

@ezYakaEagle442 Thanks for your response.

1. I see the logs from the cloud build (github workflow) and noticed the directory is empty and the build fails. But I was wondering if in your local environment some of these directories exist already. Could you run docker build locally and verify the /tmp/app directory is empty as well?

2. Could you share the steps for building the docker image locally?

I have cloned the project in order to reproduce the issue, but when running the command:
docker build --build-arg --no-cache -t "petclinic-admin-server" -f "./docker/petclinic-admin-server/Dockerfile" .

I get the following error:

[builder 7/7] RUN java -Djarmode=layertools -jar "/tmp/app/app.jar" extract:
#11 0.340 Error: Unable to access jarfile /tmp/app/app.jar

Thanks

[huanwu]

@ezYakaEagle442 I tried to repro it using az acr build. What I did is:
git clone https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/
cd to aca-java-petclinic-mic-srv
Queue a build: C:\src\aca-java-petclinic-mic-srv>az acr build --registry huanglireg -t mytest123 --file ./docker/petclinic-admin-server/Dockerfile .

I got a different error:
Step 10/25 : COPY "spring-petclinic-admin-server/target/*.jar" /tmp/app/app.jar
COPY failed: no source files were specified

I cannot find target folder under https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/tree/main/spring-petclinic-admin-server

Could you please confirm if the above repro steps are correct or not?

[ezYakaEagle442]

@huanwu @AntonioVT this is because you need to build the Java project with maven (from the project root aca-java-petclinic-mic-srv) :

mvn -B clean package --file pom.xml -DskipTests -Denv=cloud

Then you will see the Artifacts at :
aca-java-petclinic-mic-srv\spring-petclinic-admin-server\target\aca-spring-petclinic-admin-server-2.6.6.jar
aca-java-petclinic-mic-srv\spring-petclinic-api-gateway\target\aca-spring-petclinic-api-gateway-2.6.6.jar
aca-java-petclinic-mic-srv\spring-petclinic-config-server\target\aca-spring-petclinic-config-server-2.6.6.jar
aca-java-petclinic-mic-srv\spring-petclinic-customers-service\target\aca-spring-petclinic-customers-service-2.6.6.jar
aca-java-petclinic-mic-srv\spring-petclinic-vets-service\target\aca-spring-petclinic-vets-service-2.6.6.jar
aca-java-petclinic-mic-srv\spring-petclinic-visits-service\targetaca-spring-petclinic-visits-service-2.6.6.jar

[ezYakaEagle442]

here are the ref docs :

• https://learn.microsoft.com/en-us/azure/container-registry/container-registry-java-quickstart
• https://learn.microsoft.com/en-us/azure/container-registry/container-registry-troubleshoot-login#confirm-credentials-are-authorized-to-access-registry

/!\ ACR is set with Managed Identity: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication-managed-identity?tabs=azure-cli

A few steps might be missing before running az acr build command:

# /!\ ACR is set with Managed Identity: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication-managed-identity?tabs=azure-cli
# az identity create -g rg-iac-aca-petclinic-mic-srv  --name myTestIdentity
# identitySpnID=$(az identity show  -g rg-iac-aca-petclinic-mic-srv --name myTestIdentity --query principalId --output tsv)
# identityResourceID=$(az identity show  -g rg-iac-aca-petclinic-mic-srv --name myTestIdentity --query id --output tsv)
# acrID=$(az acr show -g rg-iac-aca-petclinic-mic-srv --name acrpetcliaca --query id --output tsv)
# az role assignment create --assignee $identitySpnID --scope $acrID --role acrpull
# az login --identity --username $identityResourceID 
# az acr login --name acrpetcliaca

The az login fails ()from VS/WSL, I am not connecting to a VM with :

Error detail: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/identity/oauth2/token?resource=https%3A%2F%2Fmanagement.core.windows.net%2F&api-version=2018-02-01&msi_res_id=%2Fsubscriptions%2F7b5f97dc-3c4d-424d-8288-bdde3891f242%2FresourceGroups%2Frg-iac-aca-petclinic-mic-srv%2Fproviders%2FMicrosoft.ContainerRegistry%2Fregistries%2Facrpetcliaca (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fae066d8dc0>, 'Connection to 169.254.169.254 timed out. (connect timeout=None)'))

then az acr login --name acrpetcliaca runs fine

Login Succeeded

az acr build fails with the same error message

Step 18/25 : RUN ls -al /tmp/app
 ---> Running in b9ffd5ce4bc7
total 8
drwxr-xr-x 2 root root 4096 Oct  8 09:14 .
drwxrwxrwt 1 root root 4096 Oct  8 09:14 ..
Removing intermediate container b9ffd5ce4bc7
 ---> b2e86569b20f
Step 19/25 : COPY --from=builder ${APP_INSIGHTS_AGENT_JAR_FILE_PATH} ./
 ---> f3b46f513c3e
Step 20/25 : COPY --from=builder /tmp/app/dependencies/ ./
 ---> 7f12bfbccbb5
Step 21/25 : COPY --from=builder /tmp/app/snapshot-dependencies/ ./
 ---> abd632f985c8
Step 22/25 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
failed to export image: failed to create image: failed to get layer sha256:20912aadb176614fb0b1f865eabe101834b5ad73b0fb2c796159ee6ceb32161c: layer does not exist
2022/10/08 09:14:20 Container failed during run: build. No retries remaining.
failed to run step ID: build: exit status 1

What should be implemented in the GH Workflow to sucessfully ACR build

    - name: Azure Login
      uses: azure/[email protected] # https://github.com/marketplace/actions/azure-login
      with:
        creds: ${{ env.credentials }} // This is a SPN NOT an Identity

    - name: Build image and push it to ACR
      run: |
          az acr login --name acrpetcliaca
          az acr build --registry acrpetcliaca -g rg-iac-aca-petclinic-mic-srv -t petclinic:adm-test --file "./docker/petclinic-admin-server/Dockerfile" .

[ezYakaEagle442]

@huanwu @AntonioVT you can very also tha the files are correctly extracted incuding tmp/app/sprint-boot-loader running locally :

java -Djarmode=layertools -jar aca-java-petclinic-mic-srv\spring-petclinic-admin-server\target\aca-spring-petclinic-admin-server-2.6.6.jar extract

[ezYakaEagle442]

the issue looks similar to Azure/azure-cli#20413

[AntonioVT]

Hi @ezYakaEagle442 , thanks for providing the information, I have been able to reproduce the issue.

After some investigation, we have found that this is a known issue with Moby and we have mitigation steps. There seems to be a bug when there's a sequence of COPY commands in a multi stage build. There is 1 possible workaround:

1. Adding an extra command in between two copy statements (ie: RUN true)

COPY --from=builder ${APP_INSIGHTS_AGENT_JAR_FILE_PATH} ./
COPY --from=builder /tmp/app/dependencies/ ./
COPY --from=builder /tmp/app/snapshot-dependencies/ ./
RUN true
COPY --from=builder /tmp/app/spring-boot-loader/ ./
RUN true
COPY --from=builder /tmp/app/application/ ./

The run will show logs similar to:

// Successful run from az acr build

Step 24/28 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
 ---> 75ae647c2b16
Step 25/28 : RUN true
 ---> Running in d30661d783cc
Removing intermediate container d30661d783cc
 ---> 604e692d6d3e
Step 26/28 : COPY --from=builder /tmp/app/application/ ./
 ---> 12425a941a63
Step 27/28 : EXPOSE 80 8080 8081 8082 8083 8084 8888 9090
 ---> Running in 7b0e8c1a3c00
Removing intermediate container 7b0e8c1a3c00
 ---> a553dff3cca8
Step 28/28 : ENTRYPOINT ["java", "-javaagent:/tmp/app/applicationinsights-agent-3.4.1.jar",
...   
Run ID: cc2y was successful after 1m4s

Moby issue references:
moby/moby#37965
https://stackoverflow.com/questions/51115856/docker-failed-to-export-image-failed-to-create-image-failed-to-get-layer

Please let us know if this mitigation step resolves your issue.

• Thanks @huanwu @northtyphoon for helping troubleshoot the issue offline

[ezYakaEagle442]

@AntonioVT @huanwu @northtyphoon Thank you !
I have tested locally and now the ACR build runs fine, I will test later in the GH Workflow.

FYI I did not hit this issue on ARO, which uses CRI-O , not Moby.