Deadline in Docker

A locally running render farm with Deadline.

Table of contents

• Overview
• Motivation
• Prerequsities
• Usage
  • Step I
  • Step II
  • Step III
• Submitting Jobs
• Running Arbitrary Commands
• Interactive Use
• Windows
• Deadline 9

Overview

This project proivides a locally running render farm with Deadline and is based on the article "Running Deadline 8 in Docker Part I | Part II" with the added bonus of encapsulating the entire system, including 2 slaves, via Docker Compose.

Tested on Windows and Linux.

Motivation

Developing for a distributed mechanism is akin to relying on the weather - you can never be sure what's out there or for how long it'll remain. It complicates life for a fellow developer interested in putting together a pipeline involving something like Deadline.

This project is meant as a local development silo for when you develop against Deadline, with the expectation that once you're finished you'll deploy into production and have things work the same, but without the uncertainty or surrounding infrastructure required to get Deadline off the ground.

Prerequisities

The goal of this project is to expose a private render farm for purposes of pipeline development. As such, in order to submit renders to 2 slaves running Autodesk Maya 2016, here's what (all) you need.

• Docker
• Official Deadline installer

Testing installation

This process assumes you have access to Docker via your terminal.

$ docker --version
# Docker version 17.05.0-ce, build 89658be

Usage

Use consists of (1) cloning this repository, (2) installing a Docker volume for the Deadline Repository, (3) building a Deadline client container and (4) putting it all together via Docker Compose.

Docker Compose is responsible for orchestrating the 5 pieces required for Deadline to function correctly and gluing them together accordingly.

• mongodb
• slave1
• slave2
• webservice
• samba

Step I

$ git clone https://github.com/mottosso/docker-deadline.git

Now put the official installers into the installers/ directory.

Example

docker-deadline/
  installers/
    DeadlineClient-8.0.17.2-linux-x64-installer.run
    DeadlineRepository-8.0.17.2-linux-x64-installer.run

Note that this project requires version 8 of Deadline, I didn't manage to get 9 working. See below.

Step II

Now we can go ahead and initialise the repository volume

$ cd docker-deadline
$ ./install.sh

This will install the Deadline Repository into a dedicated Docker volume called deadline-volume. Clients are then handed this volume upon startup.

You can check to see whether the volume was created correctly via docker volume.

$ docker volume list
DRIVER              VOLUME NAME
local               deadline-volume

Step III

Now let's kick things off via two machines with Maya 2016 pre-installed.

$ ./up

Pro Tip Keep in mind that each time you boot up Deadline, it'll produce a new instance of MongoDB which is where jobs are kept. The repository on the other hand is persistent as a Docker Volume. You can inspect the contents of this volume by accessing the default Samba share accessible via the IP address to the host of your container; localhost on Linux.

Step III

At this point, you'd probably want to launch Deadline Monitor.

$ ./deadlinemonitor.sh

And you're done. See Submitting Jobs for how to actually submit a job.

Submitting Jobs

Now that Deadline and all of its components are up and running, you're able to submit jobs to it via the RESTful interface.

import json
import getpass
import requests

url = "http://localhost:8082/api/jobs"
payload = {
    "JobInfo": {
        "Name": "My render",
        "UserName": getpass.getuser(),
        "Plugin": "MayaBatch",
        "Frames": "1000-1003x1",
    },
    "PluginInfo": {
        "SceneFile": "/share/my_file.ma",
        "Version": "2016",
        "OutputFilePath": "/share"
    },
    "AuxFiles": []
}

payload = json.dumps(payload)
requests.post(url, data=payload)

Under Docker Toolbox, you'll need to replace localhost with the IP to the virtual machine running your Docker Host.

You'll notice I've included an example Maya file. Once submitted, the Docker slaves will produce an image much like the one below.

Contents of Deadline Repository

You can inspect the contents of the Deadline Repository via the provided Samba container.

$ ls \\<your-ip>\\DeadlineRepository8

It will be accessible like any other shared network folder.

Pro Tip If you are running Docker Toolbox, the IP is accessible via a call to docker-machine ip.

Running Arbitrary Commands

The run.sh provides a command-line interface over the internals of the client container. It takes the following syntax.

$ ./run.sh name-of-container name-of-executable

For example

$ ./run.sh slave1 ./deadlinelauncher

Interactive Use

Gain access to the bash-prompt from within the Client container like so.

$ ./run.sh interactive bash

Restarting your farm

The farm typically runs at less than 1% of CPU capacity on my moderate, 5-year old laptop. You can kill it (gracefully) like this.

$ ./down.sh

Windows

The above assume a Linux environment. In order to run Deadline Monitor on Windows, you need a X11 server running, such as VcXsrv or Xming.

With VcXsrv running, running deadlinemonitor.sh should work well with default settings. The container is being passed $HOSTNAME from your host to represent the name with which to connect to. If this variable isn't set, you can set it your self, either to your local hostname or IP address.

$ set HOSTNAME=192.168.0.12

Deadline 9

This project containerises Deadline 8 because of an issue with the client installation under 9.

Error: An error occurred while trying to set the repository connection settings.
You may have to provide them again when running the client.

Troubleshooting

Could not connect to display

$ ./deadlinemonitor
QXcbConnection: Could not connect to display my_hostname:0
...
Deadline Monitor will now exit.

• Option A) Ensure X11 is running on your computer. Under Windows, see VcXsrv or Xming.
• Option B) Trade $HOSTNAME for your IP address in run.sh, sometimes Docker fails to resolve a hostname to an IP address

Connection Refused

Sometimes Deadline Monitor quits when running docker-compose up due to Mongo not having had enough time to initialise.

requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.99.100', port=8082): Max retries exceeded with url: /api/jobs

One solution to this is to simply bring it down, and back up again.