Support RUN --mount=type=ssh for Windows containers

[AnastaZIuk]

Hi, I've been tracking Windows containers support issue for some time and after reading this comment I wanted to check it out.

Unfortunately I have encountered issues when trying to use RUN --mount=type=ssh instruction, I'm aware Windows containers + Buildkit is experimental and in development phase so I would like to report a possible bug.

My environment is

amd64 arch Windows 11 Pro OS 10.0.22631 Build 22631
buildctl github.com/moby/buildkit v0.13.1 2ae42e0c0c793d7d66b7a23424af6fd6c2f9c8f3
containerd github.com/containerd/containerd v1.7.13 7c3aca7a610df76212171d200ca3811ff6096eb8

Reproduction

• content of dockerfile

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
USER ContainerAdministrator
RUN --mount=type=ssh ssh -T [email protected]
CMD ["cmd"]

• host commands to build an image

# setup agent with key
PS D:\test> Set-Service ssh-agent -StartupType Automatic
PS D:\test> Start-Service ssh-agent
PS D:\test> ssh-add <path/to/.ssh/id_rsa>
Identity added: <path/to/.ssh/id_rsa> (<path/to/.ssh/id_rsa>)

# build an image with ssh host proxy
PS D:\test> buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=test,push=false --ssh default
[+] Building 0.6s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                    0.0s
 => => transferring dockerfile: 174B                                                                                                                                                                    0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022                                                                                                                          0.3s
 => [internal] load .dockerignore                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                         0.0s
 => CACHED [stage-0 1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6223f3a17f2652064c56ca55aaef866177d58d6a1c526636ac2120685df5466d                                                     0.0s
 => => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6223f3a17f2652064c56ca55aaef866177d58d6a1c526636ac2120685df5466d                                                                    0.0s
 => ERROR [stage-0 2/2] RUN --mount=type=ssh ssh -T git@github.com                                                                                                                                      0.0s
------
 > [stage-0 2/2] RUN --mount=type=ssh ssh -T git@github.com:
------
Dockerfile:3
--------------------
   1 |     FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
   2 |     USER ContainerAdministrator
   3 | >>> RUN --mount=type=ssh ssh -T git@github.com
   4 |     CMD ["cmd"]
   5 |
--------------------
error: failed to solve: failed to compute cache key: failed to calculate checksum of ref okff89cu84wigb87xavgu4b8e::uohocwd1ds4k87bophhalgrag: failed to walk C:\Users\arekl\AppData\Local\Temp\buildkit-mount3862397659\WcSandboxState: open C:\Users\arekl\AppData\Local\Temp\buildkit-mount3862397659\WcSandboxState: Access is denied.
PS D:\test>

• daemon logs

PS C:\Windows\system32> buildkitd.exe
time="2024-04-12T13:46:22+02:00" level=warning msg="using null network as the default"
time="2024-04-12T13:46:22+02:00" level=info msg="found worker \"okff89cu84wigb87xavgu4b8e\", labels=map[org.mobyproject.buildkit.worker.containerd.namespace:buildkit org.mobyproject.buildkit.worker.containerd.uuid:cc938743-1aef-4599-ac57-729bed53747a org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:AnastaZIuk org.mobyproject.buildkit.worker.network: org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:windows], platforms=[windows/amd64]"
time="2024-04-12T13:46:22+02:00" level=info msg="found 1 workers, default=\"okff89cu84wigb87xavgu4b8e\""
time="2024-04-12T13:46:22+02:00" level=warning msg="currently, only the default worker can be used."
time="2024-04-12T13:46:22+02:00" level=info msg="running server on //./pipe/buildkitd"
time="2024-04-12T13:46:39+02:00" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to compute cache key: failed to calculate checksum of ref okff89cu84wigb87xavgu4b8e::uohocwd1ds4k87bophhalgrag: failed to walk C:\\Users\\arekl\\AppData\\Local\\Temp\\buildkit-mount3862397659\\WcSandboxState: open C:\\Users\\arekl\\AppData\\Local\\Temp\\buildkit-mount3862397659\\WcSandboxState: Access is denied."

I have also tried socket syntax <id>=<socket> with SSH_AUTH_SOCK env + all above with git bash shell but with no luck, I get the same errors.

[profnandaa]

@AnastaZIuk -- thanks for checking it out! are you running buildctl on an elevated PowerShell too?
nvm, I can repro it; investigating.

[AnastaZIuk]

@AnastaZIuk -- thanks for checking it out! are you running buildctl on an elevated PowerShell too? nvm, I can repro it; investigating.

yes I'm running with elevated PS, cmd behaves the same way when launched in elevated mode

[profnandaa]

Just posting here my prelimimary findings so far:

> github.com/moby/buildkit/cache/contenthash.(*cacheContext).scanPath.func2() C:/dev/container-core/buildkit/cache/contenthash/checksum.go:1031 (hits goroutine(595):1 total:1) (PC: 0x27f478a)
  1026:                 return err
  1027:         }
  1028:
  1029:         err = filepath.Walk(parentPath, func(itemPath string, fi os.FileInfo, err error) error {
  1030:                 if err != nil {
=>1031:                         return errors.Wrapf(err, "failed to walk %s", itemPath)
  1032:                 }
  1033:                 rel, err := filepath.Rel(mp, itemPath)
  1034:                 if err != nil {
  1035:                         return err
  1036:                 }
(dlv) p err
error(*io/fs.PathError) *{
        Op: "open",
        Path: "C:\\Users\\nandaa\\AppData\\Local\\Temp\\buildkit-mount1136466203\\Syst...+21 more",
        Err: error(syscall.Errno) ERROR_ACCESS_DENIED (5),}
(dlv) config max-string-len 1024
(dlv) p err.Path
"C:\\Users\\nandaa\\AppData\\Local\\Temp\\buildkit-mount1136466203\\System Volume Information"
(dlv)

I'm suspicious about the "System Volume Information" suffix in the Path.
./cc. @gabriel-samfira

[profnandaa]

I see, it's because C:\System Volume Information directory is a protected directory and can't be walked without specific ACLs. Checking how this has been treated before elsewhere or should be skipped.

[gabriel-samfira]

@profnandaa yes, this has been handled elsewhere as well, for most of the same reasons:

containerd/continuity#113
containerd/continuity#212
containerd/continuity#219

I have not looked at the ssh mount code in buildkitd, but if the containerd snapshotter is used, it should not generate this error. Will try to allocate some time to look at this this week.