local execution checkout failing #330

robeferre-leroy · 2019-09-17T21:40:16Z

I have read Contribution Guidelines.
I have checked for similar issues and haven't found anything relevant.
This is not a security issue (which should be reported here: https://circleci.com/security/)

Do you want to request a feature or report a bug?
BUG

What is the current behavior?

robeferre@mylinux:~/Workspace/golang/src/github.com/robeferre-leroy/golang-sample$ sudo circleci local execute
Docker image digest: sha256:60a98b5fc00a4302c46be2217d6e4e4eb8491d466fa62fed14362a85100bcd68
====>> Spin up Environment
Build-agent version 1.0.15283-35354d0e (2019-09-09T12:09:35+0000)
Docker Engine Version: 18.09.7
Kernel Version: Linux 6772389d44a6 5.0.0-27-generic #28~18.04.1-Ubuntu SMP Thu Aug 22 03:00:32 UTC 2019 x86_64 Linux
Starting container circleci/buildpack-deps:latest-dind
  using image circleci/buildpack-deps@sha256:f9dea38e6583c663bbe2d950b8c9abdb771f9dd587fdb18460da2f4253c3a70a

Using build environment variables
  BASH_ENV=/tmp/.bash_env-localbuild-1568756269
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=feature/circleci
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=https://github.com/robeferre-leroy/golang-sample.git
  CIRCLE_SHA1=0660ce60a8c0e3a4de9e571566ce6646a81a426a
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1568756269
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /home/circleci/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /home/circleci/project && cp -a /tmp/_circleci_local_build_repo/.git /home/circleci/project
{"Runner":true,"level":"error","msg":"Error copying logs: read /dev/ptmx: input/output error","task-id":"localbuild-1568756269","time":"2019-09-17T21:37:54Z"}
/bin/bash: line 0: cd: /tmp/_circleci_local_build_repo: No such file or directory
Error: Exited with code 1
Step failed
Error: runner failed (exited with 101)
Task failed

Can you provide an example?

robeferre@mylinux:~/Workspace/golang/src/github.com/robeferre-leroy/golang-sample$ sudo circleci local execute
Docker image digest: sha256:60a98b5fc00a4302c46be2217d6e4e4eb8491d466fa62fed14362a85100bcd68
====>> Spin up Environment
Build-agent version 1.0.15283-35354d0e (2019-09-09T12:09:35+0000)
Docker Engine Version: 18.09.7
Kernel Version: Linux 6772389d44a6 5.0.0-27-generic #28~18.04.1-Ubuntu SMP Thu Aug 22 03:00:32 UTC 2019 x86_64 Linux
Starting container circleci/buildpack-deps:latest-dind
  using image circleci/buildpack-deps@sha256:f9dea38e6583c663bbe2d950b8c9abdb771f9dd587fdb18460da2f4253c3a70a

Using build environment variables
  BASH_ENV=/tmp/.bash_env-localbuild-1568756269
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=feature/circleci
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=https://github.com/robeferre-leroy/golang-sample.git
  CIRCLE_SHA1=0660ce60a8c0e3a4de9e571566ce6646a81a426a
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1568756269
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /home/circleci/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /home/circleci/project && cp -a /tmp/_circleci_local_build_repo/.git /home/circleci/project
{"Runner":true,"level":"error","msg":"Error copying logs: read /dev/ptmx: input/output error","task-id":"localbuild-1568756269","time":"2019-09-17T21:37:54Z"}
/bin/bash: line 0: cd: /tmp/_circleci_local_build_repo: No such file or directory
Error: Exited with code 1
Step failed
Error: runner failed (exited with 101)
Task failed

What is the expected behavior?
testing builds locally

Which version of the CLI and OS are you using? Did this work in previous versions?
circleci version
0.1.5879+416032d

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"

Please provide the output of `circleci version` and `circleci diagnostic`.

CircleCI CLI Diagnostics

Debugger mode: false
Config found: /home/robeferre/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API...
Ok.
Hello, Roberto.

If you have any questions, feel free to ping us at @CircleCI-Public/x-team.

The text was updated successfully, but these errors were encountered:

DerrickRice · 2019-10-29T19:14:24Z

Same. It seems that even though --skip-checkout is the default (and even if it's explicitly mentioned), the "Checkout code" step tries to change to a directory that does not exist:

/bin/bash: line 0: cd: /tmp/_circleci_local_build_repo: Permission denied

Seems like a pretty obvious failure. Curious that this issue has not had any comments in over a month.

alexkli · 2019-11-16T02:25:46Z

I have the same issue, using 0.1.5879+416032d.

Looking at docker inspect $(docker ps -q) while circleci build is running, I can see that mounts the local working directory under /tmp/_circleci_local_build_repo inside the build container.

        "Mounts": [
            {
                "Type": "bind",
                "Source": "/Users/me/my/code",
                "Destination": "/tmp/_circleci_local_build_repo",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },

But when I do an ls /tmp/_circleci_local_build_repo (or with sudo) in my job, it says this path does not exist. So it seems the volume mounting silently fails for some reason.

Investigating further, I compared it to a manual docker run -it -v /Users/me/my/code:/tmp/_circleci_local_build_repo circleci/buildpack-deps:latest-dind /bin/bash where the mount works. The only difference I saw is when running mount, the broken one also has "none on tmpfs /tmp", which might interfere with the /tmp/... volume mount. No idea where it comes from though.

alexkli · 2019-11-16T02:59:32Z

Found a solution: manually mount the local directory using -v on circleci build. A bit of symlinking before the checkout command and it works!

config.yml snippet:

jobs:
  build:
    machine: true
    steps:
      # fix for local builds https://github.com/CircleCI-Public/circleci-cli/issues/330
      - run:
          name: Local build handling
          command: |
            if [[ ${CIRCLE_SHELL_ENV} =~ "localbuild" ]]; then
              if [ -d /workdir ]; then
                ln -s /workdir /tmp/_circleci_local_build_repo
              else
                echo "Run this local build using: circleci build -v \$(pwd):/workdir"
                exit 1
              fi
            fi
      - checkout

Then run local build using:

circleci build -v $(pwd):/workdir

notpushkin · 2020-01-10T14:29:37Z

@alexkli Thank you so much for this solution!

marcomorain · 2020-03-10T13:13:34Z

Logged as CIRCLE-25051.

…ic/circleci-cli#330 . Test env file created and its values loaded on every CI build.

Slach · 2020-04-14T12:48:34Z

@alexkli thanks for workaround
but it's not works for me
Ubuntu 18.04

 circleci local execute -c .circleci/config.yml -v .:/workdir --job build_branch
Error:   error starting container circleci/buildpack-deps:latest-dind: Error response from daemon: create [.: "[." includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path

I don't know why circleci use "[." instead of "."

manual docker run works fine
absolute path $(pwd) also not works

Unexpected environment preparation error: Error response from daemon: create [/home/test/circleci_local: "[/home/test/circleci_local" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path

maybe you can suggest workaround?

alexkli · 2020-04-14T15:59:19Z

@Slach

it must be an absolute path
looks like you run into this major regression in the latest cli version which adds [] around arguments like -v or -e: Volume Mounts for local execute started prepending [ between 0.1.6772+67c7d52 and 0.1.6949+d951526 #394 and Environment variables are not properly set #391 - you must downgrade the cli

marcomorain · 2020-05-18T13:07:57Z

I've been looking into this issue. The issue seems to be related to the image that used for the build.

Failing images:

circleci/buildpack-deps:latest-dind (the image used for "machine" builds)
circleci/rust:latest

Images that work:

cimg/base:2020.05

When we execute a build, we mount the pwd into the circleci/picard image, using --volume $pwd:$pwd. We think launch the docker container for the build with --volume $pwd:/tmp/_circleci_local_build_repo. On builds when this fails, this second mount at /tmp/_circleci_local_build_repo does not exist inside the innermost container.

BenWibking · 2020-06-14T03:48:50Z

Neither circleci/python:3.7.2 or cimg/base:2020.05 work for me. 0.1.7971+423e103 (release) and Docker version 19.03.11, build 42e35e61f3 on Ubuntu 18.04.4 LTS.

fasterthanlime · 2021-01-05T10:30:16Z

Can confirm that the simplest job fails to run locally with cimg/rust:1.49.0 (but cimg/base:2020.05 does work)

fasterthanlime · 2021-01-26T16:07:39Z

Quick update: I'm able to always reproduce this when using cimg/rust, but whenever using another base image (from the Docker Hub, or the private registry), I don't encounter this issue and I'm able to run jobs locally.

KaiyuWangCompass · 2021-02-27T01:09:52Z

Found a solution: manually mount the local directory using -v on circleci build. A bit of symlinking before the checkout command and it works!

config.yml snippet:

jobs:
  build:
    machine: true
    steps:
      # fix for local builds https://github.com/CircleCI-Public/circleci-cli/issues/330
      - run:
          name: Local build handling
          command: |
            if [[ ${CIRCLE_SHELL_ENV} =~ "localbuild" ]]; then
              if [ -d /workdir ]; then
                ln -s /workdir /tmp/_circleci_local_build_repo
              else
                echo "Run this local build using: circleci build -v \$(pwd):/workdir"
                exit 1
              fi
            fi
      - checkout

Then run local build using:

circleci build -v $(pwd):/workdir

This isn't working for me. CircleCI is not even executing this code !

kuzdogan · 2021-09-28T10:05:30Z

My local execution also fails because CircleCI checkout can't copy files with special characters. The special characters are needed in the directory name for the tests for special characters. Couldn't find a workaround.

====>> Checkout code
Making checkout directory "/home/circleci/source-verify"
Copying files from "/tmp/_circleci_local_build_repo" to "/home/circleci/source-verify"
Error: tar: "services/validation/test/files/single-add-trailing-\\n/1_Storage.sol": Cannot stat: No such file or directory
....
....

Renkas · 2023-08-28T19:45:06Z

====>> Checkout code
Making checkout directory "/home/circleci/project"
Copying files from "/tmp/_circleci_local_build_repo" to "/home/circleci/project"
Error: cp: cannot create regular file '/home/circleci/project/.git/objects/dd/f6fdf3c814d536cad1bd77b2bc86235d4da9c2': Permission denied
cp: cannot create regular file '/home/circleci/project/.git/objects/dd/56867087b589d8fa97ee544cda65a648745787': Permission denied
cp: cannot create regular file '/home/circleci/project/.git/objects/ce/acf067890b31197ce075de50b3e11bd9a29964': Permission denied
...

So what now? How can I get this to work?

Running on MacOS

alexkli mentioned this issue Nov 16, 2019

Checkout step fails on macOS #281

Closed

3 tasks

marcomorain added the bug Something isn't working label Mar 10, 2020

marcomorain added jira The issue is logged in CircleCI JIRA Local Builds Issues related to running local Docker builds. labels Mar 10, 2020

marcomorain mentioned this issue Mar 10, 2020

Access denied error in local execute #118

Closed

daerd added a commit to daerd/noisu that referenced this issue Apr 11, 2020

CircleCI local build function due to the following bug: CircleCI-Publ…

38dd6ca

…ic/circleci-cli#330 . Test env file created and its values loaded on every CI build.

learnitall mentioned this issue Jun 9, 2022

Local Execute --skip-checkout Not Respected #735

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

local execution checkout failing #330

local execution checkout failing #330

robeferre-leroy commented Sep 17, 2019 •

edited by aengelberg

Loading

DerrickRice commented Oct 29, 2019

alexkli commented Nov 16, 2019 •

edited

Loading

alexkli commented Nov 16, 2019 •

edited

Loading

notpushkin commented Jan 10, 2020

marcomorain commented Mar 10, 2020

Slach commented Apr 14, 2020

alexkli commented Apr 14, 2020

marcomorain commented May 18, 2020

BenWibking commented Jun 14, 2020

fasterthanlime commented Jan 5, 2021

fasterthanlime commented Jan 26, 2021

KaiyuWangCompass commented Feb 27, 2021 •

edited

Loading

kuzdogan commented Sep 28, 2021

Renkas commented Aug 28, 2023 •

edited

Loading

local execution checkout failing #330

local execution checkout failing #330

Comments

robeferre-leroy commented Sep 17, 2019 • edited by aengelberg Loading

Please provide the output of circleci version and circleci diagnostic.

CircleCI CLI Diagnostics

DerrickRice commented Oct 29, 2019

alexkli commented Nov 16, 2019 • edited Loading

alexkli commented Nov 16, 2019 • edited Loading

notpushkin commented Jan 10, 2020

marcomorain commented Mar 10, 2020

Slach commented Apr 14, 2020

alexkli commented Apr 14, 2020

marcomorain commented May 18, 2020

BenWibking commented Jun 14, 2020

fasterthanlime commented Jan 5, 2021

fasterthanlime commented Jan 26, 2021

KaiyuWangCompass commented Feb 27, 2021 • edited Loading

kuzdogan commented Sep 28, 2021

Renkas commented Aug 28, 2023 • edited Loading

robeferre-leroy commented Sep 17, 2019 •

edited by aengelberg

Loading

Please provide the output of `circleci version` and `circleci diagnostic`.

alexkli commented Nov 16, 2019 •

edited

Loading

alexkli commented Nov 16, 2019 •

edited

Loading

KaiyuWangCompass commented Feb 27, 2021 •

edited

Loading

Renkas commented Aug 28, 2023 •

edited

Loading