Infrastructure redesign brainstorming

[adamrehn]

This issue is partly a note to myself, but also an opportunity for other ue4-docker users to discuss potential design ideas and contribute their perspectives. Feedback from developers using ue4-docker as part of their own workflows is appreciated.

A problem I've been hitting up against for some time now is that ue4-docker was originally designed for building only a small handful of container images, and the scalability of the project is somewhat hamstrung by its current architecture. The codebase gets messier and messier as new images and customisation options are added, and also whenever new Unreal Engine versions are released that require additional logic/patching or (God forbid) a new version of Visual Studio under Windows. It's reached the point where I'm hesitant to add any of the new features that I want to implement (e.g. Perforce support, cross-compilation support in Windows containers, hooks that allow custom Dockerfile instructions to be injected by the user at various points, etc.) because I can just picture how convoluted the code will look if those modifications are added to the existing design. It's time for a refactor and a new approach.

My current line of thought is that ue4-docker should split the build process into two distinct phases: an initial generation phase that produces a set of Dockerfiles on disk and then a build phase that consumes the generated Dockerfiles. This would facilitate a template-based approach whereby Dockerfiles can be customised with a high degree of flexibility, without littering the resulting code with no-ops for unused options (as is the case in the current architecture.) This could be paired with a plugin-based system for applying arbitrary transformations to the template data, allowing for an infinite number of possible configurations. I'm also toying with the idea of using a graph-based system for managing dependencies between container images, which could open up the door for user-defined images that declaratively specify their dependencies and get built alongside the predefined images in the correct order. Regardless of the exact shape that the implementation takes, my primary goal is to create a data-oriented architecture with a clean and well-abstracted API.

A few benefits that I foresee:

• Newer (or older) versions of Visual Studio can be supported by simply providing drop-in snippets for each version that can be selected by the user at generation-time.
• The addition of cross-compilation toolchains (e.g. Windows to Linux, Windows to Android) can be implemented a clean manner that doesn't impact images generated without cross-compilation support.
• If users want to implement support for new version control systems (Perforce, Mercurial, etc.), authentication mechanisms (SSH, OAuth, etc.) or even to copy the Unreal Engine source tree from the host filesystem (as in Pull Request #47), they can do so by simply implementing a plugin, eliminating the need to modify ue4-docker itself. Popular plugins could of course ship with ue4-docker for ease of installation.
• Injection of arbitrary user-specified Dockerfile directives can be achieved either via plugins or simply by editing the generated Dockerfiles prior to building them.
• Users who need to maintain a large set of images (e.g. game development studios who are cross-compiling for multiple platforms) can script the generation step, or even check the generated Dockerfiles into source control.

A few challenges and open questions:

• An infinite number of possible configurations makes thorough testing extremely difficult, leading to an increased likelihood of users encountering issues when building configurations that I've not tried myself.
• How can we tag images and manage inter-image dependencies in a sensible manner? Ideally all customisations should manifest themselves as suffixes in image tags so that downstream images can specify the required configuration for their base images. But with an infinite number of possible configurations, any one-to-one mapping could produce ridiculously long and unwieldy tag names.
• What's the most reasonable way to embed configuration metadata in built images so that they can be inspected after being pulled from a container registry? (Hopefully this scenario will become a reality once GitHub Package Registry comes out of beta.) Is it sensible to simply create LABEL directives specifying each configuration option, such that a hypothetical ue4-docker inspect command could simply query the labels to print a summary of the configuration that produced the image? If so, do we require all plugin authors to specify label names for their customisations and follow a reverse-DNS naming scheme to avoid collisions? How should this account for manual modifications to generated Dockerfiles, if at all?

This redesign is likely to remain an ongoing process for some time to come, and I'd appreciate feedback from other ue4-docker users as I evolve these ideas and move towards a point where I can begin implementing them.

[adamrehn]

@ImmortalEmperor, @mentalnote, @crizCraig, @TBBle, @herman-rogers

[TBBle]

A use-case I've been meaning to follow-up with here, would be to have a plugin for BuildGraph changes. In our particular use-case, we sometimes need a branch of GitHub UE4 Release tags only to add the BuildGraph commands for the UE4Client and UE4Server targets to the InstalledEngineBuild.xml and InstalledEngineFilter.xml.

I'd also planned to try and work out how to make them conditional, and get those added to Epic's upstream version, but in the meantime it'd be nice to auto-apply them to any random UE4 source tree I build. That auto-applying might need per-version fixing, and I'd aim to make it generic enough to apply to all platforms, at least for the UE4Client target.

I had planned to propose this as a patch to this project if and when I had time, but a plugin architecture would make it much cleaner, I think.

[TBBle]

On LABEL, presumably a plugin must have some identification so it's loaded, simply using that in the LABEL would work, wouldn't it? It seems reasonable to expect plugins to provide a label for images generated with that plugin, and even strongly encourage labels to be reverse-DNS to avoid name collisions and provide some sense of providence for the plugin, if all you you have at hand is the resulting Docker image.

[adamrehn]

@TBBle that's definitely a use-case that's been on my mind too, and you're not the only one to run into the lack of support for those targets by default. It seems odd that there isn't already a BuildGraph flag to enable support for them, but I guess it's yet another symptom of the fact that Epic don't consume Installed Builds internally.

[adamrehn]

With regards to LABEL directives, that's certainly the line of thought I've been considering. Requiring each plugin to specify a unique reverse-DNS identifier would be extremely helpful for diagnostic and debugging purposes, since ue4-docker could list the loaded plugins in its output and those lines could then be included when reporting bugs. (A maintainer email address might also be a good idea so that the appropriate plugin maintainers can be contacted when dealing with bugs arising from plugin code.)

If plugins are able to specify multiple feature flags (which might be desirable in a context where flags are related, or are mutually incompatible and the plugin needs to throw an error when a conflict is detected) then each individual flag could simply be appended to the plugin's identifier, e.g.:

LABEL com.organisation.plugin.flag=1

[treilhes]

Hi adamrehn,
First of all thanks for this awesome project.
Second, take a look at https://github.com/EpicGames/UnrealEngine/commit/b8622b1f. It seems like Server and Client target support is back in 4.23

Third, if i understand correctly you are aiming to create a Dockerfile(s) generator where plugins will provide short templates which will be processed and inserted in resulting Dockerfile(s)? I think it is a realy good idea but it will need a separate project to be successfull. The generator must be generic without any ties to ue4-docker. ue4-docker will use this project providing some kind of descriptor (json/xml/...) and a "UE4-DOCKER plugin".

A first step would be to support legacy docker commands as plugin (RUN/COPY/FROM) or a generic PRINT plugin to translate your existing images to a descriptor. Something like:

{
  image: "ue4-build-prerequisites",
  qualifier: "linux",
  build: [
    { plugin: "PRINT", args: ["ARG BASEIMAGE"] },
    { plugin: "PRINT", args: ["FROM ${BASEIMAGE}"] },
    { plugin: "PRINT", args: ["LABEL com.adamrehn.ue4-docker.sentinel="1""] },
    { plugin: "PRINT", args: ["ENV DEBIAN_FRONTEND=noninteractive"] },
    ....
    { plugin: "PRINT", args: ["USER ue4"] },
  ],
  child-images : [
    {
      image: "ue4-source",
      qualifier: "linux",
      file-dependencies: [
        "git-credential-helper.sh",
        "linker-fixup.py",
        ...
      ],
      build: [ .... ],
      child-images : [
        {
          image: "ue4-minimal",
          qualifier: "linux",
          build: [ 
             .... ,
             { plugin: "UE4DOCKER", command: "RunUAT", args: [.....] },
             ...
          ]
        },
        {
          image: "ue4-full",
          qualifier: "linux",
          build: [ .... ]
          ]
        },
      ]
    }
  ]
}

Then progressively include other features:

• variable management
• conditional execution
• ...

It is a project in itself. Maybe looking into existing templating system may do the trick.

[treilhes]

Another possible approach may be to use Ansible playbooks to build the images and create an ue4-docker-ansible module

[adamrehn]

@treilhes awesome, thanks for pointing out that commit! It's good to see that those targets finally have official support.

Yeah, I'm still investigating the different ways I could go about implementing the new system. My current line of thinking is that plugin logic will be written in Python, but I have no concrete ideas about what form the declarative aspects of the system (such as the template fragments and image dependency specifications) will take at this point. I agree that it'll need to be separate from the legacy version of ue4-docker until it reaches the stage where the old implementation can be retired.

[dskvr]

What is the status on this refactor initiative?

[adamrehn]

@dskvr this is currently blocked by an upstream limitation regarding support for building Windows containers with BuildKit (moby/buildkit#616) that makes it impossible to mount secrets during the build process without them leaking into the image's filesystem layers, as is possible when building Linux container images using the experimental mount types supported by BuildKit.

One of the driving factors behind the proposed two-phase design is that it will address the current incompatibilities with external build systems, which are typically designed to accept ordinary Dockerfiles (and optionally, accompanying build arguments) as their inputs, but are not designed to delegate control of the build to an arbitrary application. In the new design, ue4-docker will generate a set of Dockerfiles in filesystem directories (similar to standard base images found on Docker Hub, e.g. Microsoft .NET, Node.js, etc.) and the user can store those files / check them into version control / etc. and then build them later without requiring ue4-docker at all. For this to be feasible, the credential endpoint needs to be eliminated entirely, as it is currently the only component of ue4-docker that is strictly required during the build process. This is currently possible for Linux containers but not Windows containers, and providing a harness to run the credential endpoint when building Windows containers defeats the intended purpose. (Additional harnesses would also need to be created for supporting SSH credentials or Perforce credentials, and community-contributed plugins for the generation phase that use those authentication types would be forced to integrate with the harnesses under Windows, which isn't a scalable strategy.)

This is definitely still a high priority, and we're going to be needing this internally at TensorWorks as we start exploring support for targeting console platforms in on-premises installations of Admiral (since I'm guessing those target platforms will likely require cross-compilation from Windows rather than Linux.) There's a good chance that I'll end up continuing the work that @TBBle started on adding functional Windows container support to BuildKit, since fixing the upstream limitation seems like the only way to achieve the two-phase design with feature parity across both Windows and Linux containers.

[dskvr]

@adamrehn Thanks! What are the chances these changes will be public before Admiral is complete?

[adamrehn]

@dskvr any changes made to ue4-docker (or any of my other open source projects) will always be public as soon as they're implemented and working well enough for other users to benefit from them. The only thing I won't be able to make public are any plugins that specifically relate to console platforms, since there are legal restrictions around that, but everything else will be available for the community as soon as it's working nicely across both Windows and Linux containers.

[TBBle]

I do intend to spend more time on getting BuildKit working on Windows, but I haven't had the good stretches of time available for a personal project in a while.

Plus it kind-of got stuck on whether I needed to support containerd 1.2 as a back-end with the changes I was make, e.g. moby/buildkit#1387, and I lost a lot of momentum on that and containerd/containerd#2366, as I was hoping for some kind of upstream movement before I pushed too much further down a certain direction.

So what I pushed to my fork is the latest I have, which worked-around or ignored those issues, and there's plenty that can be done in areas which do not relate to them, like the networking setup, the filesystem-copy/permissions issue, etc.

I suspect getting it upstreamed will be equally as hard as actually getting it working, in the end.

[adamrehn]

Quick update: although @TBBle's valiant efforts are slowly turning the tide, parity between Windows containers and Linux containers is still looking to be rather distant, and as such so is the refactor. In the interim, I've decided to implement a subset of the intended functionality of the refactor (specifically being able to generate Dockerfiles that can be built without ue4-docker) for Linux containers so that users and downstream projects that stand to benefit from this feature can start doing so while we wait for our blocking upstream limitations.

To this end, I'll be adding a -layout DIR flag to the ue4-docker build command which will copy the generated Dockerfiles to the specified directory. (And yes, I appreciate the irony of my choice to borrow this terminology from the analogous flag in the Visual Studio installer under Windows when this feature will only be of practical use for Linux containers in the immediate future.) With this in place, I will then add an option to leverage BuildKit's support for bind-mounting secrets when building Linux containers, removing the generated Dockerfiles' dependency on the ue4-docker credential endpoint and thus allowing them to be built without requiring the use of ue4-docker itself.

Note that I'm not planning on implementing the other features of the refactor yet (user plugins and Dockerfile customisation) since those represent much larger problems that cannot be solved without fundamentally restructuring the way that ue4-docker is implemented. Those features will still need to wait for the actual refactor when it eventually happens.

[adamrehn]

These changes are now live in ue4-docker version 0.0.79 and are documented here:

• Exporting generated Dockerfiles
• Advanced options for Dockerfile generation

It's nowhere near the level of customisation that I'll be aiming for in the refactor, but at least it's something that everyone can make use of in the meantime.

[adamrehn]

I've now provided a more comprehensive write-up on the new features, which includes an example of using them: https://adamrehn.com/articles/preview-the-future-of-ue4-docker/