Failure building in Windows Container due to missing file

[dkmartov]

Hi!

ue4-docker version:         0.0.72 (latest available version is 0.0.72)
Operating system:           Windows 10 Pro Version 1903 (OS Build 18362.1139)
Docker daemon version:      19.03.13
NVIDIA Docker supported:    No
Maximum image size:         200GB
Available disk space:       168.46 GiB
Total system memory:        31.95 GiB physical, 41.26 GiB virtual
Number of processors:       6 physical, 12 logical

Hi I'm attempting to run the command

ue4-docker build custom:MYTAG -repo=MYREPO -branch=release

However I get a failure building with the message. I believe this is the relevant snippet of my log:

Saving local manifest to C:\UnrealEngine\Engine\Saved\BuildGraph\Build Tools Win64\Manifest.xml
ERROR: Attempt to add file to temp storage manifest that does not exist (C:\UnrealEngine\cpp.hint)
(see C:\UnrealEngine\Engine\Programs\AutomationTool\Saved\Logs\Log.txt for full exception trace)
AutomationTool exiting with ExitCode=1 (Error_Unknown)
BUILD FAILED
The command 'cmd /S /C .\Engine\Build\BatchFiles\RunUAT.bat BuildGraph -target="Make Installed Build Win64" -script=Engine/Build/InstalledEngineBuild.xml -set:HostPlatformOnly=true -set:WithDDC=%BUILD_DDC% && del /s /q C:\UnrealEngine\LocalBuilds\InstalledDDC 2>NUL && python C:\exclude-components.py C:\UnrealEngine\LocalBuilds\Engine\Windows %EXCLUDE_DEBUG% %EXCLUDE_TEMPLATES% && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.' returned a non-zero code: 1
[ue4-docker build] Error: failed to build image "adamrehn/ue4-minimal:martov-1903"

As you can see it appears to be attempting to add a file that does not appear to exist. I am not sure if this is because of my windows setup.

Do you have any idea why this is happening?

Also I am not sure how to get the full log short of piping into a file but I attached the ending section that wasn't cropped out by the terminal. Please let me know if you can, where to retroactively find the log.

ue4-dockerlog.txt

[dkmartov]

Small update: I ran a diagnostic on my system. It does suffer for the 8gb filesystem issue.
That said I reran the command and got the same error.

[dkmartov]

New update. I tried the same command on windows server 2019 and got the same error
Here is the info for the machine

ue4-docker version: 0.0.72 (latest available version is 0.0.72)
Operating system: Windows Server 2019 Datacenter Version 1809 (OS Build 17763.1518)
Docker daemon version: 19.03.12
NVIDIA Docker supported: No
Maximum image size: 200GB
Available disk space: 702.91 GiB
Total system memory: 15.82 GiB physical, 18.69 GiB virtual
Number of processors: 4 physical, 8 logical

Is there something I am doing wrong??
Thanks.

here is the error from the new log.
Creating library C:\UnrealEngine\Engine\Intermediate\Build\Win64\UnrealFrontend\Development\MetalShaderFormat\UnrealFrontend-MetalShaderFormat.suppressed.lib and object C:\UnrealEngine\Engine\Intermediate\Build\Win64\UnrealFrontend\Development\MetalShaderFormat\UnrealFrontend-MetalShaderFormat.suppressed.exp
UnrealFrontend.target
Saving local manifest to C:\UnrealEngine\Engine\Saved\BuildGraph\Build Tools Win64\Manifest.xml
ERROR: Attempt to add file to temp storage manifest that does not exist (C:\UnrealEngine\cpp.hint)
(see C:\UnrealEngine\Engine\Programs\AutomationTool\Saved\Logs\Log.txt for full exception trace)
AutomationTool exiting with ExitCode=1 (Error_Unknown)
BUILD FAILED
The command 'cmd /S /C .\Engine\Build\BatchFiles\RunUAT.bat BuildGraph -target="Make Installed Build Win64" -script=Engine/Build/InstalledEngineBuild.xml -set:HostPlatformOnly=true -set:WithDDC=%BUILD_DDC% && del /s /q C:\UnrealEngine\LocalBuilds\InstalledDDC 2>NUL && python C:\exclude-components.py C:\UnrealEngine\LocalBuilds\Engine\Windows %EXCLUDE_DEBUG% %EXCLUDE_TEMPLATES% && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.' returned a non-zero code: 1

[TBBle]

You haven't mentioned what version of UE4 your tree is branched from.

I don't see any failures in the build log. I'm not sure what is supposed to generate cpp.hint, but it's possible that there was a failure before the start of the log that's supposed to generate cpp.hint, but somehow didn't stop the build when it failed.

That said, try increasing the storage-opt to 400GB or 600GB and see if that fixes it (per #112). I'd also suggest adding --exclude debug as at the end of this step, you're likely to hit #44 during filesystem commit of this stage.

Interestingly, 4.26 removes the reference to cpp.hint from this build step: https://github.com/EpicGames/UnrealEngine/commit/006feb2ef072d41fa429c585031c50955c3c2ba5, so possibly this is a known-flakey thing.

[dkmartov]

Thanks. Its just a vanilla fork of UE release branch ( which corresponds to 4.25.4). No code changes.

I did already attempt the following things to the same result:
1.) I increased the storage-opt to 600gb
2.) I added --exclude debug and tried running the command again.

Unfortunately I get the same error. I'm going to try building again and saving full log to see if there is an error generating this file. Afterwards I'm going to try building a version of 4.26 instead. Thanks for response.

[arisona]

I'm running into the same issue, also with 4.26 - any resolution / fix to this (my max image size is 800GB)?

[TBBle]

4.26 is also complaining about missing "C:\UnrealEngine\cpp.hint"? That's very strange, since 4.26 removed that file from the buildsystem.

Can you provide the full build log please, so we can see how it got into this situation in the first place?

[arisona]

I was wrong, apologies - ignore my last post, with 4.26 I did not get the missing file issue, but it aborted with copy error. I then tried patching the dockerfile according to #99 (comment), but still getting it.

I'm on Windows Server 2019 Standard Version 1809, with Docker server 19.03.12. The 8GB diagnostic fails on this system, so could be related to this?

I'm letting it run again and will post the output.

[TBBle]

It's probably best to follow up on #99 if you're having the COPY error, so that information about that issue stays in one place, and we can keep this ticket for the 'cpp.hint' issue. For example, the workaround might need to be updated for 4.26.

[tynril]

FWIW, the error with the missing cpp.hint file was a bug in the 4.25.4 release, which was fixed on Epic's Perforce release branch, but not on GitHub (the 4.25.4-release tag doesn't include the fix).

This is the commit that addressed the bug in the initial 4.25.4 release: https://github.com/EpicGames/UnrealEngine/commit/84e4ea3241c294c04fdf7d8fb63f99a3109c8edd

This is a bug with UE's build scripts themselves, rather than with the environment. Presumably, applying that patch to Engine/Build/Commit.gitdeps.xml would fix ue4-docker's ability to build 4.25.4. Similarly, if you're using a custom engine repo, you can backport that patch to 4.25.4 to get around the problem.

[TBBle]

Great to know. So @adamrehn or @dkmartov, I'd propose to close this ticket as "Upstream bug in 4.25.4 GitHub release: Backport the fix into your own UE4 tree, use 4.25.3, or wait for 4.26 to be released".

[dkmartov]

Upstream bug in 4.25.4 GitHub release: Backport the fix into your own UE4 tree, use 4.25.3, or wait for 4.26 to be released

[adamrehn]

I'm used to .0 releases of new Unreal Engine versions having bugs in them that subsequently get fixed, but this is definitely the first time I've seen the final point release of a version introduce a bug that is then left unfixed until the next .0 release. Since I always advise people to use the newest point release of any given Engine version, I've added a patch in commit 99ca1b0 that injects the fix from https://github.com/EpicGames/UnrealEngine/commit/84e4ea3241c294c04fdf7d8fb63f99a3109c8edd when it detects version 4.25.4 and the fix isn't already present (so it won't interfere with people's own forks that already include the backport.)

I'll tag a new release of ue4-docker that includes the new change once I've tested that it fixes the build issues under both Windows and Linux.

[adamrehn]

My tests builds were successful for 4.25.4 under both Windows and Linux. The fix is now live in ue4-docker version 0.0.73.