ue4-source doesn't fail (and ends up missing files) if runs out of disk space during dependencies download

[jonpas]

Output of the ue4-docker info command:

PS C:\Users\Administrator> ue4-docker info
ue4-docker version:         0.0.81 (latest available version is 0.0.82)
Operating system:           Windows Server 2019 Standard Version 1809 (OS Build 17763.1852)
Docker daemon version:      20.10.0
NVIDIA Docker supported:    No
Maximum image size:         400GB
Available disk space:       448.11 GiB
Total system memory:        8 GiB physical, 9 GiB virtual
Number of processors:       4 physical, 4 logical

Additional details:

• Are you accessing the network through a proxy server? No

Building vanilla ue4-minimal 4.26.1 on Windows Server 2019 fails BuildGraph with AutomationTool (see log below).

Building is happening in a Windows VM (process-isolation) with 160 GB SSD and 500 GB HDD (attached block storage). Docker is installed on the HDD so it has enough space to build on that one.

PS C:\Users\Administrator> ue4-docker build 4.26.1 --no-engine -username jonpas -password SNIP
[ue4-docker build] COMMAND-LINE INVOCATION:
[ue4-docker build] ['ue4-docker', '4.26.1', '--no-engine', '-username', 'jonpas', '-password', '*******']

[ue4-docker build] UNREAL ENGINE VERSION SETTINGS:
[ue4-docker build] Custom build:  No
[ue4-docker build] Release:       4.26.1
[ue4-docker build] Repository:    https://github.com/EpicGames/UnrealEngine.git
[ue4-docker build] Branch/tag:    4.26.1-release

[ue4-docker build] WINDOWS CONTAINER SETTINGS
[ue4-docker build] Isolation mode:               process
[ue4-docker build] Base OS image tag:            ltsc2019 (host OS is Windows Server version 1809)
[ue4-docker build] Memory limit:                 No limit
[ue4-docker build] Detected max image size:      400GB
[ue4-docker build] Directory to copy DLLs from:  C:\Windows\System32

[ue4-docker build] GENERAL SETTINGS
[ue4-docker build] Not excluding any Engine components.
[ue4-docker build] Not building the ue4-source image, no Git credentials required.

[ue4-docker build] Image "adamrehn/ue4-build-prerequisites:ltsc2019" exists and rebuild not requested, skipping build.

[ue4-docker build] Image "adamrehn/ue4-source:4.26.1-ltsc2019" exists and rebuild not requested, skipping build.

[ue4-docker build] User specified `--no-engine`, skipping ue4-engine image build.

[ue4-docker build] Building image "adamrehn/ue4-minimal:4.26.1-ltsc2019"...
Sending build context to Docker daemon  15.87kB
Step 1/27 : ARG NAMESPACE
Step 2/27 : ARG TAG
Step 3/27 : ARG PREREQS_TAG
Step 4/27 : FROM ${NAMESPACE}/ue4-source:${TAG}-${PREREQS_TAG} AS builder
 ---> b72a3a87ad21
Step 5/27 : RUN rmdir /s /q C:\UnrealEngine\.git && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 55e72906a4c9
Step 6/27 : COPY set-changelist.py C:\set-changelist.py
 ---> Using cache
 ---> 977435f1e5b6
Step 7/27 : RUN python C:\set-changelist.py C:\UnrealEngine\Engine\Build\Build.version && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 40bd590b3305
Step 8/27 : COPY patch-filters-xml.py C:\patch-filters-xml.py
 ---> Using cache
 ---> 67c1e19a8253
Step 9/27 : RUN python C:\patch-filters-xml.py C:\UnrealEngine\Engine\Build\InstalledEngineFilters.xml && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 1d6d9bd3d966
Step 10/27 : COPY patch-build-graph.py C:\patch-build-graph.py
 ---> Using cache
 ---> c0de8b3b975a
Step 11/27 : RUN python C:\patch-build-graph.py C:\UnrealEngine\Engine\Build\InstalledEngineBuild.xml && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Using cache
 ---> 16011d14f708
Step 12/27 : ARG BUILD_DDC
 ---> Using cache
 ---> c6027510f8e9
Step 13/27 : WORKDIR C:\UnrealEngine
 ---> Using cache
 ---> 5a5c480b77f8
Step 14/27 : COPY exclude-components.py C:\exclude-components.py
 ---> Using cache
 ---> 4fc785d89adf
Step 15/27 : RUN .\Engine\Build\BatchFiles\RunUAT.bat BuildGraph -target="Make Installed Build Win64" -script=Engine/Build/InstalledEngineBuild.xml -set:HostPlatformOnly=true -set:WithDDC=%BUILD_DDC%  &&     rmdir /s /q C:\UnrealEngine\LocalBuilds\InstalledDDC 2>NUL && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.
 ---> Running in 556f4de5d0d8
Running AutomationTool...
Parsing command line: BuildGraph -target="Make Installed Build Win64" -script=Engine/Build/InstalledEngineBuild.xml -set:HostPlatformOnly=true -set:WithDDC=true -compile
Dependencies are out of date. Compiling scripts....
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "AWSSDK.Core". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "AWSSDK.S3". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(3,7): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(4,7): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(5,7): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(6,7): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(597,32): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(702,29): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(714,34): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(742,29): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(754,34): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(170,19): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "BouncyCastle.Crypto". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\IOS\IOS.Automation.csproj]
Took 10.5375425s to run MSBuild.exe, ExitCode=1
ERROR: Failed to build "C:/UnrealEngine/Engine/Programs/AutomationTool/Saved\UATTempProj.proj":
       C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "AWSSDK.Core". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "AWSSDK.S3". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(3,7): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(4,7): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(5,7): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(6,7): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(597,32): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(702,29): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(714,34): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(742,29): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(754,34): error CS0246: The type or namespace name 'AmazonS3Client' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\UploadDDCToAWS.cs(170,19): error CS0246: The type or namespace name 'Amazon' could not be found (are you missing a using directive or an assembly reference?) [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\Scripts\AutomationScripts.Automation.csproj]
       C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "BouncyCastle.Crypto". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\IOS\IOS.Automation.csproj]

       (see C:\UnrealEngine\Engine\Programs\AutomationTool\Saved\Logs\Log.txt for full exception trace)
AutomationTool exiting with ExitCode=1 (Error_Unknown)
BUILD FAILED
The command 'cmd /S /C .\Engine\Build\BatchFiles\RunUAT.bat BuildGraph -target="Make Installed Build Win64" -script=Engine/Build/InstalledEngineBuild.xml -set:HostPlatformOnly=true -set:WithDDC=%BUILD_DDC%  &&       rmdir /s /q C:\UnrealEngine\LocalBuilds\InstalledDDC 2>NUL && echo. && echo.RUN directive complete. Docker will now commit the filesystem layer to disk. && echo.Note that for large filesystem layers this can take quite some time. && echo.Performing filesystem layer commit... && echo.' returned a non-zero code: 1
[ue4-docker build] Error: failed to build image "adamrehn/ue4-minimal:4.26.1-ltsc2019".

ue4-docker diagnostics all passes without issues.

[slonopotamus]

It looks like something has gone wrong during build of ue4-source image. Could you please try removing it and retrying again?

[jonpas]

I will attempt this again in a few weeks most likely.

[jonpas]

So having another attempt at this I found what the issue was. Updating dependencies in step 19/22 in ue4-source image failed due to missing space. The error was just printed but the build continued and "succeeded". Of course, it was missing lots of files, so compilation in ue4-minimal straight-up failed with the above errors.

This was a direct result of me moving Docker's data-root to a 2nd drive which has enough space compared to the main C drive. This resulted in ue4-docker setup to only update the main one as seen here, as well as ue4-docker info reporting everything correct, as that just reads that same file as well (found that here).

I used #99 to diagnose the space issue and discovered it stopped just shy of 20GB. which is the default limit. Manually setting the limit in the new data-root's daemon.json of course fixed the issue.

I am not sure if there is another way to move Docker to another drive or if there is a way to provide or specify Docker's data-root location to ue4-docker, for setup and info. That line in the source code suggests it is not.

---

To summarize, I see 2 possible improvements here.

1. Provide a way to set data-root location.
2. Fail the image build if step 19/22 in ue4-source fails to retrieve all dependencies.

[slonopotamus]

Manually setting the limit in the new data-root's daemon.json of course fixed the issue.

Err... I'm not sure I understand you. On Windows, there is C:\ProgramData\Docker\config\daemon.json. If you want to use custom data-root, you specify it in C:\ProgramData\Docker\config\daemon.json. But there doesn't exist a second daemon.json inside data-root (because if there is one, you've got chicken-and-egg problem: in order to find daemon.json, you need to know data-root, but data-root is set inside daemon.json). Is you setup somehow different?

On the second point (Fail the image build if step 19/22 in ue4-source fails to retrieve all dependencies) I totally agree with you.

[jonpas]

Yes, you can change location of data-root another way, by unregistering and registering the dockerd service using dockerd --register-service --data-root e:\docker. Using it this way actually creates a fully new Docker environment, with its own daemon.json.

This may not be the ideal way to do it depending on use-case, but it seems to be one of the supported ways to do it.