(maint) Enable Docker builds with buildkit

[Iristyle]

This is an experiment to see if we can improve Docker build times by using buildkit. Buildkit will enable a few things:

• Using multiple targets inside of a single Dockerfile and smart evaluation (i.e. no need to have multiple Dockerfiles like base, release, etc)
• Concurrent build processes where possible, rather than strictly sequential
• Better caching (not applicable to Travis, but very applicable to Windows / LCOW)

It looks like this is generally a no-go at this point in time due to the LCOW error, which could be a permissions issue (though I tested setting Everyone: (F) for C:\ProgramData\docker\tmp and C:\ProgramData\docker\windowsfilter and didn't see anything stand out in procmon):

failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to mount C:\ProgramData\docker\tmp\buildkit-mount984162069: [{Type:bind Source:C:\ProgramData\docker\windowsfilter\8es1az6gkgzbc4u4w031q6maz Options:[rbind ro]}]: invalid windows mount type: 'bind'

This is something we probably want to evaluate again in the future (or help fix under LCOW) - buildkit currently says Windows is unsupported - moby/buildkit#616

Update: It looks like it may be possible to use buildkit more directly rather than going through docker build. buildctl has a Windows binary and it's possible to run the buildkit daemon in a container.

That's a reasonably different workflow from what we're doing now, but seems like it may plausibly work without much modification. The convention for passing command line args would vary a bit. For instance, --build-arg vcs_ref=$(vcs_ref) would become --opt build-arg:vcs_ref=$(vsc_ref). See https://github.com/moby/buildkit#exploring-dockerfiles for more details.

[Iristyle]

Looks like this might not be possible on Windows (at least not easily):

docker build --build-arg version=6.7.2 --build-arg vcs_ref=3c27f646dc61db97a94a8f6f73795b00f0fcdb77 --build-arg build_date=--file docker/puppetserver-base/Dockerfile --memory 4g --build-arg namespace=puppet --tag puppet/puppetserver-base:6.7.2 --pull docker/puppetserver-base
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 2.62kB done
#1 DONE 0.2s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.2s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to mount C:\ProgramData\docker\tmp\buildkit-mount519259336: [{Type:bind Source:C:\ProgramData\docker\windowsfilter\fvor99jmpaj39d3lp5ljauvuu Options:[rbind ro]}]: invalid windows mount type: 'bind'
##[error]PowerShell exited with code '1'.
##[section]Finishing: Build puppetserver-base Container

[Iristyle]

Travis docker version is super old, and buildkit isn't supported here either.

docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:11:02 2018
 OS/Arch:           linux/amd64
 Experimental:      false
Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:09:05 2018
  OS/Arch:          linux/amd64
  Experimental:     false

It's possible to upgrade with https://docs.travis-ci.com/user/docker/#installing-a-newer-docker-version

[Iristyle]

I think it's probably quicker to just pull the packages from https://download.docker.com/linux/ubuntu/dists/xenial/pool/stable/amd64/ and install that way rather than using the PPA. The Xenial PPA doesn't have the latest builds for some reasons anyway!

[Iristyle]

Nope, not true. Verified that the time is about the same - except the code to install packages is a fair amount more complicated (especially when resolving dependencies).

Better to use the PPA

[Iristyle]

Update timings (after a few successful runs through)... looks like there is some time to save on builds right now (30s?), even when the host appears slower for other steps.

before_install 0.64s -> 26.36s / 43.62s
before_script 350.44s -> 320.47s / 317.77s
script - 141.40s -> 131.81s / 141.19s

[Iristyle]

Not sure if it makes sense to only upgrade the Docker PPA - i.e. something like

apt-get update -o Dir::Etc::sourcelist=/path/to/repo.list

https://askubuntu.com/a/698028

[Iristyle]

Buildkit support was merged in #2328

This seems to have been left open by accident