DQ Minecraft Server tech docs

The server is running on AWS EC2 in a dedicated t3a.medium instance:

• Amazon Linux 2
• 2-thread 2.5 GHz
• 4 GB RAM
• 100 GB SSD
• Physically in eu-central-1a zone, that's in Frankfurt

The world is 700 MB in size, the map is 12 GB. Plus we store 10 last hourly world backups on-site. 100 GB of space should be plenty for a while.

AWS billing is managed by @deltaidea.

Map rendering

Enter --render-map in the game chat and wait for a while. You'll see progress messages in the chat.

If there're no automated messages, you're not alone, this happens from time to time. Message @deltaidea or fix it yourself:

• Download the SSH key from Trello (see the tech details card) and save it to ~/.ssh
• Log into the VM using ssh -i ~/.ssh/minecraft-ec2.pem [email protected]
• Run screen -S minecraft_watch_commands -X quit to kill the command detection loop
• Run this to restart it:

  screen -dmS minecraft_watch_commands ~/scripts/minecraft_watch_commands.py
  screen -S minecraft_watch_commands -X logfile ~/minecraft_watch_commands_process.log
  screen -S minecraft_watch_commands -X log

• Quit the VM using exit
• Try writing --render-map in the game chat again

It takes about 10-15 minutes to render the map. Here's what the rendering script does specifically:

• Spins up a separate insanely powerful EC2 instance using AWS API
• Makes a world backup and copies it over to the renderer instance.
• Installs Overviewer over there and downloads our map config: https://github.com/dq-server/overviewer-config
• Renders the map (this step takes 8-12 minutes)
• Syncs the map to the main instance
• Terminates the rendering instance because it costs $5 an hour to run

Logs and troubleshooting

The following log files may be helpful if something goes wrong:

• ~/minecraft/logs/latest.log
• ~/minecraft_process.log
• ~/minecraft_watch_commands_process.log
• ~/map_process.log
• ~/management_api_process.log

Relevant commands:

• tail --lines 50 <file> - shows the last 50 lines of a file. Log files can have thousands of lines, usually you only need to see the latest events.
• ls -lhAF <directory> - shows directory contents with human-readable sizes, including hidden files.
• scp -r -i ~/.ssh/minecraft-ec2.pem [email protected]:~/minecraft/logs ./ - run this locally on your machine to download all MC logs.
• sudo systemctl stop map && sudo systemctl start map - restart the map HTTP server if you can't load the map in your browser.
• sudo systemctl stop minecraft && sudo systemctl start minecraft - restart the Minecraft server if something goes horribly wrong.
• uptime - see VM uptime and average CPU load for the last 1, 5, 15 minutes.
• top - task manager. Quit by pressing q.
• last - shows login history. See who logged into the VM before you and when.

Backups

We're running two backup strategies:

• Hourly backups on-site - to a folder next to the Minecraft server. Only the last 10 hourly backups are stored.
• Weekly backups off-site - to AWS Glacier. They're stored forever.

To recover from an hourly backup, log into the server (ssh -i ~/.ssh/minecraft-ec2.pem [email protected]) and run:

# Look at the dates and choose a suitable backup
ls -lhAF ~/minecraft-backups
# Change the argument to the backup number, in this case we're using backup-5
~/scripts/minecraft_restore.sh 5

You can also download an hourly backup to your PC by running the following command on your machine:

scp -r -i ~/.ssh/minecraft-ec2.pem [email protected]:~/minecraft-backups/backup-0 ./world-backup

To recover from a weekly off-site backup, message @deltaidea. Guide: https://docs.aws.amazon.com/cli/latest/reference/glacier/initiate-job.html

Shutting down and restarting the server

• To safely shut down the server, enter --system-shutdown in the game chat.
• To start the server, open https://manage.minecraft.deltaidea.com and enter the access key from Trello.

Cron and autorun

We have three systemctl services running on the server: one for Minecraft, Nginx server for the map, and one for DynDNS updating. They all start automatically after a reboot.

minecraft.service also spawns a separate process that watches game logs to respond to our custom commands described above.

There're three cron jobs: two for backups and another one for LetsEncrypt cert refreshing for our management API.

HTTP Management API

The server has a custom little HTTP API, see management_api.py. It runs as a systemctl service, see management_api.service. It's available at https://minecraft.deltaidea.com:5000

This API is used by manage.minecraft.deltaidea.com to get Minecraft status. Minecraft only has TCP API, and JavaScript in browsers doesn't allow to send raw TCP packets. So in order to see Minecraft status on the page we have to get it from some kind of a backend. That's why we have this Python script.

There're two endpoints:

• GET /minecraft-status returns JSON with the game version and a list of players online. It calls ./minecraft_get_status.sh internally to get this info.
• GET /map-status returns {"status": 200} after making a request to 127.0.0.1/overviewer.js which should be successful if the map isn't down.

There's no authentication at this point, though it would be neat to have some kind of access control for potential future features like server shutdown or map refreshing.

SSL

Because manage.minecraft.deltaidea.com uses HTTPS, JavaScript on that page can't make unencrypted HTTP requests. So to make requests to the management API, it has to have SSL enabled. We're using Let's Encrypt for that. You don't have to back up any credentials or anything, this is just FYI. See Migration guide for more info about how it's installed.

Migrating to another machine

If there's ever a need to migrate to a new machine, we've documented the installation process in migration_guide.md.