buildet: read-only directory in work area leads to cleanup failure when not root

[jclulow]

When a buildlet is running as a non-root user, read-only directories like those managed by the Go module system can cause the work directory cleanup step to fail. The buildlet ends up exiting, and the stage0 program goes into a loop like this:

stage0: 2019/10/18 05:41:18 bootstrap binary running
stage0: 2019/10/18 05:41:18 waiting for network.
stage0: 2019/10/18 05:41:18 network up after 400ms
stage0: 2019/10/18 05:41:18 downloading https://storage.googleapis.com/go-builder-data/buildlet.illumos-amd64 to ./buildlet.exe ...
stage0: 2019/10/18 05:41:18 downloaded ./buildlet.exe (14194957 bytes)
stage0: 2019/10/18 05:41:18 downloaded buildlet in 100ms
2019/10/18 05:41:19 buildlet starting.
2019/10/18 05:41:19 unlinkat /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/[email protected]/LICENSE: permission denied
stage0: 2019/10/18 05:41:19 Error running buildlet: exit status 1
...

The file we're trying to remove is read-only:

[root@gobuild1 ~]# stat /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/[email protected]/LICENSE
  File: /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/[email protected]/LICENSE
  Size: 15921           Blocks: 18         IO Block: 16384  regular file
Device: 5a000109c5h/386547124677d       Inode: 125098      Links: 1
Access: (0444/-r--r--r--)  Uid: (84001/ gobuild)   Gid: (84001/ gobuild)
Access: 2019-10-18 04:37:10.739769295 +0000
Modify: 2019-10-18 04:37:10.740269133 +0000
Change: 2019-10-18 04:37:10.740269133 +0000
 Birth: 2019-10-18 04:37:10.739769295 +0000

But the real issue is that the directory in which the file appears is also read-only:

[root@gobuild1 ~]# stat /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/[email protected]/
  File: /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/[email protected]/
  Size: 14              Blocks: 3          IO Block: 131072 directory
Device: 5a000109c5h/386547124677d       Inode: 125094      Links: 3
Access: (0500/dr-x------)  Uid: (84001/ gobuild)   Gid: (84001/ gobuild)
Access: 2019-10-18 04:37:10.738867957 +0000
Modify: 2019-10-18 04:37:10.741754528 +0000
Change: 2019-10-18 04:37:10.742657507 +0000
 Birth: 2019-10-18 04:37:10.738867957 +0000

According to unlinkat(2):

       The unlink() and unlinkat() functions will fail if:

       EACCES
                       Search permission is denied for a component of the path
                       prefix, or write permission is denied on the directory
                       containing the link to be removed.

You can see this with a relatively simple test case:

#!/bin/bash
  
set -o errexit
set -o pipefail

ROOT=/var/tmp/blahblah

chmod -R u+rwx "$ROOT" || true
rm -rf "$ROOT"

mkdir "$ROOT"
mkdir "$ROOT/first"
mkdir "$ROOT/first/second"
date > "$ROOT/first/second/file0"
date > "$ROOT/first/second/file1"
mkdir "$ROOT/first/third"
date > "$ROOT/first/third/file2"
date > "$ROOT/first/third/file3"
chmod 0444 "$ROOT/first/second/file0"
chmod 0444 "$ROOT/first/third/file2"
chmod 0500 "$ROOT/first/third"

find "$ROOT" -ls

package main

import (
        "os"
        "log"
)

func main() {
        rootdir := "/var/tmp/blahblah"

        if err := os.RemoveAll(rootdir); err != nil {
                log.Fatalf("removing \"%s\": %v", rootdir, err)
        }

        log.Printf("ok, removed \"%s\"", rootdir)
}

$ go build trial.go 
$ ./testcase.sh 
91262    1 drwxr-xr-x   3 jclulow  nfsusers        3 Oct 18 06:54 /var/tmp/blahblah
91263    1 drwxr-xr-x   4 jclulow  nfsusers        4 Oct 18 06:54 /var/tmp/blahblah/first
92547    1 dr-x------   2 jclulow  nfsusers        4 Oct 18 06:54 /var/tmp/blahblah/first/third
92548    1 -r--r--r--   1 jclulow  nfsusers       36 Oct 18 06:54 /var/tmp/blahblah/first/third/file2
92549    1 -rw-r--r--   1 jclulow  nfsusers       36 Oct 18 06:54 /var/tmp/blahblah/first/third/file3
92544    1 drwxr-xr-x   2 jclulow  nfsusers        4 Oct 18 06:54 /var/tmp/blahblah/first/second
92545    1 -r--r--r--   1 jclulow  nfsusers       36 Oct 18 06:54 /var/tmp/blahblah/first/second/file0
92546    1 -rw-r--r--   1 jclulow  nfsusers       36 Oct 18 06:54 /var/tmp/blahblah/first/second/file1
$ ./trial 
2019/10/18 06:54:20 removing "/var/tmp/blahblah": unlinkat /var/tmp/blahblah/first/third/file2: permission denied

As root, though, this succeeds:

$ pfexec ./trial
2019/10/18 06:55:15 ok, removed "/var/tmp/blahblah"

Prior to calling os.RemoveAll(), the buildlet should make an attempt to ensure all directories under the work area are writeable.

[gopherbot]

Change https://golang.org/cl/201917 mentions this issue: cmd/buildlet: handle read-only directories in work area