Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The hcsshim on Windows 10 1903 always fails to build Docker image #624

Closed
eriawan opened this issue Jun 11, 2019 · 47 comments
Closed

The hcsshim on Windows 10 1903 always fails to build Docker image #624

eriawan opened this issue Jun 11, 2019 · 47 comments

Comments

@eriawan
Copy link

eriawan commented Jun 11, 2019 •
edited
Loading

After I have updated my machine from Windows 10 1809 to 1903, I always experiencing errors when doing image build.

The error always shows that the problem is caused by hcsshim:

F:\dockerimages>docker build --rm -f Dockerfile -t dev/testimage .
Sending build context to Docker daemon  330.6MB
Step 1/11 : FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-windowsservercore-ltsc2016
 ---> d529d6d68c79
Step 2/11 : SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
 ---> Running in 3a7c4fcc808c
hcsshim::PrepareLayer failed in Win32: Incorrect function. (0x1) path=C:\ProgramData\Docker\windowsfilter\3a7c4fcc808c8794606111ba9fb67342560aa5bfa40a888d564e52871fbd8840

image

This issue is always reproducible on Windows 10 1903, and on previous version of Windows 10 it doesn't happen. I have tested on other machines that have WIndows 10 1803 and Windows 10 1809, and using the same Docker for Windows Desktop (2.0.0.3), the image build is always finished/done successfully.

This has been reported before on Docker for Windows: docker/for-win#3884 especially this comment and the next comments after:
docker/for-win#3884 (comment)
@eriawan
Copy link
Author

eriawan commented Jun 17, 2019

@jhowardmsft @jterry75 @jstarks

Is this repo active? This issue has been 6 days without any reply.
Please confirm...

@jbolduan
Copy link

I'm also getting this error since upgrading my machine to 1903 when trying to build any dockerfile (that I've tried so far).

It seems to get to the first config step and then fail.

@gatsby0121
Copy link

Was anyone able to fix this? I'm getting it as well.

@eriawan eriawan mentioned this issue Jul 1, 2019
Closed
@thaJeztah
Copy link
Contributor

/cc @ddebroy

@xuelix
Copy link

xuelix commented Jul 3, 2019

Just updated my Win 10 to 1903. Seeing the same issue.

@moo-ecxio
Copy link

moo-ecxio commented Jul 4, 2019 •
edited
Loading

got the same issue on win10 release 1903 when trying to copy content from a container to the file system:
docker cp [containerName]:[containerPath] [hostFilesystem]
resulting in
Error response from daemon: hcsshim::PrepareLayer failed in Win32: Incorrect function. (0x1) path=C:\ProgramData\Docker\windowsfilter\6a7019f5530c15951e6a7441dba27ed40a87c560b4ab5befe324aa26956c498d

Docker version 18.09.2, build 6247962

@eriawan
Copy link
Author

eriawan commented Jul 4, 2019

@jterry75 @jhowardmsft

Guys, please reply/response soon.
Is this repo actively monitored for issues too? Is it only active for code changes (commits) only?
Because I have seen active commits, but on issues are not having responses at all.

Please confirm, as this issue is now aged more than 3 weeks.... 😞

As for me, I think I'm going to rollback my Windows to be 1809 and I won't update to 1903 to build any docker container on Windows 10 1903 until this issue is solved, and this is quite a showstopper for me and my teams, including other folks that have experienced this issue too.

@giniyat202
Copy link

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images.
Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason.
I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags.
It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds.
USE AT YOUR OWN RISK

@antonkhelou
Copy link

After upgrading to 1903, I couldn't build anything. The only thing that fixed it was removing the storage-ops.

@thaJeztah
Copy link
Contributor

@antonkhelou that's storage-ops in the daemon configuration? What storage-ops did you have configured?

@thaJeztah
Copy link
Contributor

Oh, I see a comment in moby/moby#36831 (comment)

If you have a different storage location configured, could you try if things work if you disable Windows defender, or exclude the location? https://support.microsoft.com/en-us/help/4028485/windows-10-add-an-exclusion-to-windows-security

@giniyat202
Copy link

giniyat202 commented Jul 12, 2019 •
edited
Loading

I'm using the default storage location, and not using storage-opts
Changing the storage location, excluding the storage location and disabling Windows defender won't solve the problem in my case.
EDIT: using storage-opts results in a slightly different error Error response from daemon: hcsshim::PrepareLayer failed in Win32: The parameter is incorrect. (0x57) when using the hack in my earlier comment. when not using the hack, the error is not changed.

@lowenna
Copy link
Contributor

lowenna commented Jul 12, 2019

This is not an hcsshim issue. I'm closing this. See docker/for-win#3884 (comment)

@lowenna lowenna closed this as completed Jul 12, 2019
@lyling
Copy link

lyling commented Aug 6, 2019

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images.
Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason.
I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags.
It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds.
USE AT YOUR OWN RISK

how to use it?

@giniyat202
Copy link

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images.
Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason.
I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags.
It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds.
USE AT YOUR OWN RISK

how to use it?

First you should compile the program. I created a c++ console application on Visual Studio 2015 but generally any c compiler that targets Win32 should work.
After that, you can use an elevated powershell to run the following command:
\path\to\generated\executable.exe p (ps dockerd).id.
You can also revert the changes by running \path\to\generated\executable.exe u (ps dockerd).id.

@thaJeztah thaJeztah mentioned this issue Aug 6, 2019
Merged
@mitchcapper mitchcapper mentioned this issue Oct 3, 2019
Closed
@Stannieman Stannieman mentioned this issue Oct 6, 2019
Closed
@lyling
Copy link

lyling commented Oct 21, 2019 •
edited
Loading

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images.
Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason.
I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags.
It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds.
USE AT YOUR OWN RISK

how to use it?

First you should compile the program. I created a c++ console application on Visual Studio 2015 but generally any c compiler that targets Win32 should work.
After that, you can use an elevated powershell to run the following command:
\path\to\generated\executable.exe p (ps dockerd).id.
You can also revert the changes by running \path\to\generated\executable.exe u (ps dockerd).id.

@giniyat202
It cant work again. it works follow your workaround a few days ago. but after i do some reset docker desktop, update windows. i found i cant work. and i can`t track what different.

should i compile it again after windows update ? now it is windows 10 1903(18362.418)

@giniyat202
Copy link

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images.
Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason.
I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags.
It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds.
USE AT YOUR OWN RISK

how to use it?

First you should compile the program. I created a c++ console application on Visual Studio 2015 but generally any c compiler that targets Win32 should work.
After that, you can use an elevated powershell to run the following command:
\path\to\generated\executable.exe p (ps dockerd).id.
You can also revert the changes by running \path\to\generated\executable.exe u (ps dockerd).id.

@giniyat202
It cant work again. it works follow your workaround a few days ago. but after i do some reset docker desktop, update windows. i found i cant work. and i can`t track what different.

should i compile it again after windows update ? now it is windows 10 1903(18362.418)

@lyling I have just updated it for version 10.0.18362.175 and it should work as usual once you recompile it

@thaJeztah
Copy link
Contributor

Looks like regression is also tracked in #708, and a workaround has been merged in #718

Work is in progress to get that workaround merged into the Docker codebase (see moby/moby#40128)

@lyling
Copy link

lyling commented Oct 25, 2019

@giniyat202 you have deleted the simple program ????

@giniyat202
Copy link

@giniyat202 you have deleted the simple program ????

no, its still available

@lyling
Copy link

lyling commented Oct 28, 2019

@giniyat202 you have deleted the simple program ????

no, its still available

@giniyat202 , i get the code and i see you do an update for 418. it works great. thank you so much.

@dayue-liang
Copy link

dayue-liang commented Nov 2, 2019 •
edited
Loading

hello,

I compiled simple-program using MinGW-w64.
I executed a program but return error.

C:\docker\fix>dockerd-patch.exe p 8096
CreateToolhelp32Snapshot failed
What is wrong ??

@lyling
Copy link

lyling commented Nov 2, 2019 via email

@dayue-liang
Copy link

Thanks! > lyling

I used command prompt by administrator and failed.
And then I used powershell by administrator and succeeded.

@rioam2
Copy link

rioam2 commented Nov 8, 2019

@giniyat202 Any insight on how I can migrate this patch to work for 18362.449? I've noticed that you updated the check_integrity_bytes in your previous update.

@giniyat202
Copy link

@giniyat202 Any insight on how I can migrate this patch to work for 18362.449? I've noticed that you updated the check_integrity_bytes in your previous update.

I use x64dbg for that.
The exact steps required to update this patch are as follows:

  1. Load vmcompute.dll inside x64dbg. You can find this file under %systemroot%\system32.
  2. Select Symbols tab, right click vmcompute.dll module and select Download symbols for this module . You also need to make note of Base address next to the module.
  3. Type ?MountVhd in the search bar below the right pane then double click the only result
  4. Hit Ctrl-r
  5. At this point it is required to find the call that passes 376 on edx, In my case it was the 4th reference.
  6. Assuming nothing has changed in the function body, check_integrity_offset should be calculated as by subtracting Base obtained in step 2 from the address of xor r8d, r8d instruction and check_integrity_bytes are the bytes starting from this instruction all the way to the ja instuction slightly below the call

I will update my code once I receive the update (Windows update didn't offer it for some reason)

@rioam2
Copy link

rioam2 commented Nov 8, 2019

@giniyat202 Thank you! Turns out that your script still works with 18362.449, and my issue was just user error. Assuming Microsoft doesn't fix anytime soon, these instructions will be very handy for the next breaking update 👍

@El-Gor-do
Copy link

El-Gor-do commented Nov 11, 2019 •
edited
Loading

@giniyat202 I'm trying to update the patch for 18363.449 but I'm having problems following your instructions. After opening vmcompute.dll, the Symbols tab doesn't show vmcompute.dll. The left panel shows dllloader64_3158.exe, apphelp.dll, msvcp_win.dll, ucrtbase.dll, gdi32full.dll, kernelbase.dll, win32u.dll, gdi32.dll, user32.dll, kernel32.dll and ntdll.dll. How do I download symbols for vmcompute.dll?

@giniyat202
Copy link

@giniyat202 I'm trying to update the patch for 18363.449 but I'm having problems following your instructions. After opening vmcompute.dll, the Symbols tab doesn't show vmcompute.dll. The left panel shows dllloader64_3158.exe, apphelp.dll, msvcp_win.dll, ucrtbase.dll, gdi32full.dll, kernelbase.dll, win32u.dll, gdi32.dll, user32.dll, kernel32.dll and ntdll.dll. How do I download symbols for vmcompute.dll?

You might have set your debugger to stop at system breakpoint. Try to resume the program (F9) and see if it becomes visible.
also @rioam2 mentions that no modification is required for 18363.449 so you might want to try the current version first.

@El-Gor-do
Copy link

El-Gor-do commented Nov 11, 2019 •
edited
Loading

@giniyat202 thanks for your help, I managed to figure it out now. It turns out that the check_integrity_bytes and check_integrity_offset for 18363.449 is identical to 18362.418. Just to expand on your instructions for those of us who are not x64dbg experts,

2 - If vmcompute.dll doesn't appear in the modules list then keep pressing F9 until vmcompute.dll appears.

5 - In the References tab, double click on each reference to jump to the call in the CPU tab, then scroll up 3 lines and check if the preceding 3 instructions are "xor r8d,r8d", "mov edx,376" and "mov rcx,qword ptr ss:[rbp-58]"

@El-Gor-do
Copy link

El-Gor-do commented Nov 11, 2019 •
edited
Loading

After all this, it turns out that I haven't been reading the Docker error message correctly. My hcsshim error is 0x57 rather than 0x1 so this patch isn't applicable to my case. Doh!

@giniyat202
Copy link

After all this, it turns out that I haven't been reading the Docker error message correctly. My hcsshim error is x57 rather than 0x1 so this patch isn't applicable to my case. Doh!

do you have storage-ops in your config? If so try to remove it and see if it works

@El-Gor-do
Copy link

I didn't have storage-opts set at all in my daemon.json. I also tried with "storage-opts": [ "size=20479M" ] but still no luck so I guess I just have to wait for Microsoft to fix the bug.

@bplasmeijer
Copy link

bplasmeijer commented Nov 13, 2019 •
edited
Loading

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images.
Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason.
I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags.
It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds.
USE AT YOUR OWN RISK

how to use it?

First you should compile the program. I created a c++ console application on Visual Studio 2015 but generally any c compiler that targets Win32 should work.
After that, you can use an elevated powershell to run the following command:
\path\to\generated\executable.exe p (ps dockerd).id.
You can also revert the changes by running \path\to\generated\executable.exe u (ps dockerd).id.

Using the patch with storage-opt NOT working;

I did try the patch but from error code hcsshim::PrepareLayer - failed failed in Win32: The parameter is incorrect. (0x01) to hcsshim::PrepareLayer - failed failed in Win32: The parameter is incorrect. (0x57)

without the storage-opt setting, docker build works fine!

(https://github.com/imphasing/dockerPatcher)

@giniyat202 windows version 1903 (18362.418)

@mitchcapper
Copy link

0x57 is well documented in: #708 has been patched in hcsshim with a work around already for windows 1903. The work around being merged into docker can be found in terms of progress at: moby/moby#40128 once merged in a new release will have to be made. 0x01 error has also already been patched and merged: moby/moby#39679 you will just need a binary newer than that for that fix (or wait for the backport release docker-archive/engine#314 binaries).

@eriawan
Copy link
Author

eriawan commented Nov 13, 2019

Hi guys!

After ensuring these steps:

  1. uninstalling Docker, deleting Docker data,
  2. deleting/uninstalling online storage desktop (such as Box, DropBox)
  3. having fresh install of Docker Desktop 2.1.0.4 with ensuring that I have no storage-opts in the Docker configuration,

Now I have successful Docker build!

I have this configuration:

  • Windows 10 1903 build 18362.476
  • Docker Desktop 2.1.0.4 (39773)
  • Docker Engine 19.03.4

Here's my Docker about box:

image

I have tried to apply the suggested patch from @giniyat202 , but the patch didn't work for me.

NOTES:

  1. Uninstall all of the online drive storage that permeates as having virtual file system, such as Box, Dropbox on your machine that is used to build Docker image.
  2. The only one online drive storage that has nothing to do with this issue is Microsoft OneDrive. (don't ask me why, I really don't know 😞 )
  3. Apparently before I have Windows 10 1903 updates of build 18362.449, this issue still happened. It seems something from Microsoft has fixed this issue (on Windows 10 side).
  4. I have no problem running Docker build on both Windows 10 builds of 18362,449 and 18362.476, as long as I have at least Docker Desktop 2.1.0.4 (detail in the about box above).

Feel free to tell me if my suggestion doesn't work for some or all of you.

@El-Gor-do
Copy link

El-Gor-do commented Nov 14, 2019 •
edited
Loading

I've just updated Windows to 18363.476 and I am now able to build Windows images without setting "storage-opts": [ "size=20G" ]

Edit... actually, no. "docker-compose build" still fails with 0x57 but "docker build ." is able to build a Windows image. So my workaround for "docker-compose build" is "docker build -t ." to build my Windows image first, then run "docker-compose build" which will successfully use docker-compose.yml to create my container.

@bplasmeijer
Copy link

deleting/uninstalling online storage desktop (such as Box, DropBox) will solve the problem @El-Gor-do

@El-Gor-do
Copy link

@bplasmeijer I don't use Box or DropBox but I found that uninstalling OneDrive (which comes preinstalled with Windows 10) fixed the 0x57 error when using docker-compose.

@El-Gor-do
Copy link

Looks like I spoke too soon, After uninstalling OneDrive then I was able to docker-compose but after rebooting, 0x57 occurs again despite no longer having OneDrive installed.

@wawa0210
Copy link

wawa0210 commented Nov 28, 2019 •
edited
Loading

After upgrading to 1909 and the docker version is 19.03.5 and the same problem when i run image with storage-opt param,such as

image

Any one have the same problems?

@jseixas01
Copy link

Hello,
Always the same problem: "hcsshim::PrepareLayer - failed failed in win32 : Incorrect function. (0x1)"

Docker Engine 19.03.5
Docker Desktop 2.1.0.5
Windows 10 1903 build 18362.535

Configuration file:
{
"registry-mirrors": [],
"insecure-registries": [],
"debug": true,
"experimental": false,
],
"data-root": "E:\Docker"
}

I have always the same error with or without "storage-opts" and also when using default location.
I am nw on Docker am I need urgently to solve this issue.

Can You help me ?

Thanks.

João

@DDieterich
Copy link

Windows just did an update and laid down a new copy of this driver.

  1. Rename the C:\Windows\System32\drivers\cbfs6.sys as something else (or delete it).
  2. Remove the container.
  3. Reboot.

See: docker/for-win#3884 (comment)

@kaidokert
Copy link

Running on Windows Version 10.0.18362, patch provided by @giniyat202 worked perfectly. Thanks a lot ! Also thanks to @imphasing for putting in a VS solution

@Bhodges94
Copy link

Bhodges94 commented Oct 26, 2021 •
edited
Loading

Hi,
I tried following the steps above and got as far as compiling this C++ program and then it threw some check integrity error and followed @giniyat202 instructions on x64dbg but when you get to the last step it doesn't actually say what to do with the information.

I'm on Windows 10 20H2, have run windows update and have the latest docker (4.1.1)

The error I get seems the same. I don't know what to do with this check integrity error from the C++ app so bit lost what i can do now.

Sending build context to Docker daemon 5.632kB
Step 1/2 : FROM mcr.microsoft.com/windows/nanoserver:1809
---> 1dba00abbe16
Step 2/2 : USER ContainerAdministrator
---> Running in 7da904ee3667
hcsshim::PrepareLayer - failed failed in Win32: Incorrect function. (0x1)

thanks
*update, I have upgraded to the latest windows 10 version (21H1) and still same issue
I've just uninstalled box as per @El-Gor-do suggestion and restarted and it now works

@xakod
Copy link

xakod commented Nov 24, 2021

Problem still here

docker pull mcr.microsoft.com/windows/nanoserver:1909
1909: Pulling from windows/nanoserver
fe8c39bf0334: Extracting [==================================================>]  102.1MB/102.1MB
failed to register layer: re-exec error: exit status 1: output: hcsshim::ProcessBaseLayer \\?\C:\ProgramData\Docker\windowsfilter\c93bfe644648f253e1aedf8d036ef5ba16436fdb59097fe5c97eba860b6dbe59: The parameter is incorrect.

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.3)
  compose: Docker Compose (Docker Inc., v2.1.1)
  scan: Docker Scan (Docker Inc., 0.9.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.10
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 19043 (19041.1.amd64fre.vb_release.191206-1406)
 Operating System: Windows 10 Pro Version 2009 (OS Build 19043.1348)
 OSType: windows
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.881GiB
 Name: CIBuilder
 ID: ***
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Is there any fix? dockerd patcher cannot work because it doesn't find vmcompute.dll.

@Antonin750
Copy link

Antonin750 commented Apr 15, 2022 •
edited
Loading

Hi,
The patch works fine for Windows 11 with a new version of vmcompute.dll.
Obviously you need to correctly find the new edx position. Be careful the check_integrity_bytes changed too.

I don't know why that doesn't work with the 376 value and that must be replace by another one but great job

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests