Guacamole instances randomly freezing

[jsf9k]

🐛 Bug Report

Guacamole instances running recently-built AMIs randomly freeze. When this happens

• All ssh sessions freeze
• New ssh sessions cannot be established
• Guacamole sessions freeze

To Reproduce

Steps to reproduce the behavior:

• Instantiate an instance from a recently-built AMI generated from this repository
• Wait long enough, and it will freeze

Expected behavior

Such instances should not freeze.

[jsf9k]

I tried tailing logs and looking at dmesg output while waiting for such an instance to freeze, but time and time again I saw nothing of interest. Clearly the freeze was taking place before the logs could output anything. Therefore the issue looks very much like a kernel panic.

[jsf9k]

Hoping that this issue was indeed caused by a kernel panic, I manually enabled kernel crash dumps in running Guacamole instances. I basically used these instructions, although I found some of the steps listed there to be superfluous. What is actually required is:

1. Modify the Guacamole server's security group and network ACLs to allow port 80 outbound.
2. sudo apt-get install kdump-tools $(uname -r)-dbg
3. Append nmi_watchdog=1 to the kernel command line in /etc/default/grub
4. sudo update-grub
5. Reboot the instance

[jsf9k]

There are some reports online of kernel panics appearing. See, for instance, these links:

• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=966846
• https://forum.proxmox.com/threads/kernel-5-4-44-causes-system-freeze-on-hp-microserver-gen8.72050/page-2
• https://lkml.org/lkml/2020/7/20/883

[jsf9k]

We caught a kernel panic on env0 in our staging environment. I added the information to the original Debian bug report, since it looks like our problem is indeed a result of the recent kernel changes.

Links for investigating kernel crash dumps using crash:

• https://www.linuxjournal.com/content/oops-debugging-kernel-panics-0
• https://www.thegeekstuff.com/2014/05/kdump/

[jsf9k]

It took several failed attempts (partly because doing kernel foo in Docker is not a great way to test), but I modified the Guacamole AMI to boot into an older kernel that we used before and know to be good. See these PRs:

• Pin debian kernel ansible-role-upgrade#11
• Remove existing kernels for Debian 10 ansible-role-upgrade#12
• Do not remove all kernel headers ansible-role-upgrade#13
• Configure grub to use the desired kernel ansible-role-upgrade#14

Once the kernel bug has been resolved and is available in Debian, we can undo these changes.

[jsf9k]

It is worth noting that this issue is probably appearing in the Guacamole instance because we are running Docker there and the kernel bug involves cgroups.

[jsf9k]

Resolved in the pRs listed above, as well as #24.