Refine concurrency logic (#5063)

* Fix for unchecked concurrency

We need to make sure we're not using so many cores that the tests
are starved of memory.

Signed-off-by: Adam Farley <[email protected]>

* Adjusting memory size math to compare bytes with bytes

Instead of what it does right now, which is to compare bytes with
kilobytes, and incorrectly assume that the kilobytes number is always
smaller.

Signed-off-by: Adam Farley <[email protected]>

* Updating memory limit check to allow cgroup V2 compatability

In cgroup v1 we used memory.limit_in_bytes to store the maximum
memory allocated to the container. In v2, we use memory.max. This
change allows us to check for both, includes a meminfo check for
non-containers, and adds a few debug comments so we can be sure this
new code is working.

My plan is to run tests on a diverse set of machines, and to remove
the debug statements prior to merging.

Signed-off-by: Adam Farley <[email protected]>

* Include docker check and cgroup version check

To ensure we're looking in the correct file for the maximum memory
size. Also to handle permissions issues, empty files, etc.

Signed-off-by: Adam Farley <[email protected]>

* Exporting debug message

Signed-off-by: Adam Farley <[email protected]>

* Switching back to file existance checks for simplicity

Makes it easier to maintain, and more resilient to failure.

I've also added a commented, formatted copy of the script to help
people read it in the future.

Signed-off-by: Adam Farley <[email protected]>

* Correcting comment symbol. Typo.

Signed-off-by: Adam Farley <[email protected]>

* Switching to curley braces to access variable value

Signed-off-by: Adam Farley <[email protected]>

---------

Signed-off-by: Adam Farley <[email protected]>

openjdk/openjdk.mk

@@ -20,7 +20,37 @@ ARCH:=$(shell uname -m)
 
 ifeq ($(OS),Linux)
 	NPROCS:=$(shell nproc)
-	MEMORY_SIZE:=$(shell KMEMMB=`awk '/^MemTotal:/{print int($$2/1024)}' /proc/meminfo`; if [ -r /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then CGMEM=`cat /sys/fs/cgroup/memory/memory.limit_in_bytes`; else CGMEM=`expr $${KMEMMB} \* 1024`; fi; CGMEMMB=`expr $${CGMEM} / 1024`; if [ "$${KMEMMB}" -lt "$${CGMEMMB}" ]; then echo "$${KMEMMB}"; else echo "$${CGMEMMB}"; fi)
+	# This is the MEMORY_SIZE script below, with formatting for readability.
+	#
+	# // The number of megabytes of memory this machine has.
+	# KMEMMB=`awk '/^MemTotal:/{print int($$2/1024)}' /proc/meminfo`; 
+	#
+	# // If this machine/container uses cgroups to limit the amount of 
+	# // memory available to us, we should use that as out memory size.
+	# if [[ -r /sys/fs/cgroup/memory.max ]]; then
+	#     // Use this to identify memory maximum (bytes) for cgroup v2.
+	#     CGMEM=`cat /sys/fs/cgroup/memory.max 2>1`; 
+	# else
+	#     // Else use this file for memory maximum (bytes) on cgroup v1.
+	#     CGMEM=`cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>1`; 
+	# fi; 
+	#
+	# // If those files were empty, or didn't exist, or had non-numbers
+	# // in them, then use /proc/meminfo (converted to bytes).
+	# if [[ ! $$(CGMEM) =~ ^[0-9]+$$ ]]; then
+	#     CGMEM=`expr $${KMEMMB} \* 1024 \* 1024`; 
+	# fi; 
+	#
+	# CGMEMMB=`expr $${CGMEM} / 1024 / 1024`; 
+	#
+	# // Between memory limits in the cgroup and memory on the machine,
+	# // use the lower limit. This protects us against situations
+	# // where the cgroup has a value which is much bigger/smaller than
+	# // the limits on the machine overall. We've seen both.
+	# if [ "$${KMEMMB}" -lt "$${CGMEMMB}" ]; then 
+	#     echo "$${KMEMMB}"; else echo "$${CGMEMMB}"; 
+	# fi
+	MEMORY_SIZE:=$(shell KMEMMB=`awk '/^MemTotal:/{print int($$2/1024)}' /proc/meminfo`; if [[ -r /sys/fs/cgroup/memory.max ]]; then CGMEM=`cat /sys/fs/cgroup/memory.max 2>1`; else CGMEM=`cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>1`; fi; if [[ ! $${CGMEM} =~ ^[0-9]+$$ ]]; then CGMEM=`expr $${KMEMMB} \* 1024 \* 1024`; fi; CGMEMMB=`expr $${CGMEM} / 1024 / 1024`; if [ "$${KMEMMB}" -lt "$${CGMEMMB}" ]; then echo "$${KMEMMB}"; else echo "$${CGMEMMB}"; fi)
 endif
 ifeq ($(OS),Darwin)
 	NPROCS:=$(shell sysctl -n hw.ncpu)