wmstats - consistent reporting of wmagent disk use

[mapellidario]

Fixes #12139

Status

Tested. I run this code inside a wmagent docker container and i got the expected output [1].

Description

the output of df is

(WMAgent-2.3.5) [cmst1@vocms0262:~]$ df -klP
Filesystem     1024-blocks    Used Available Capacity Mounted on
/dev/vdb         257837508 3991660 240722264       2% /etc/group

At the moment in WMStats we show the value of the latest column "mounted on", which is not meaningful nor consistent across agents. The first column "filesystem" contains the name of the partition that we care about and is consistent both between the docker container and the host and across different hosts.

This PR adds the parsing of the first column and shows that string in wmstats

Is it backward compatible (if not, which system it affects?)

YES: I did not remove the "mounted on" column, so existing code will continue to work as is

Related PRs

none

External dependencies / deployment changes

none

---

[1]

(WMAgent-2.3.5) [cmst1@vocms0262:~]$ vim Utilities.py
(WMAgent-2.3.5) [cmst1@vocms0262:~]$ python
Python 3.8.16 (default, May 23 2023, 14:26:40)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import Utilities
>>> Utilities.diskUse()
[{'filesystem': 'overlay', 'mounted': '/', 'percent': '22%'}, {'filesystem': 'tmpfs', 'mounted': '/dev', 'percent': '0%'}, {'filesystem': 'shm', 'mounted': '/dev/shm', 'percent': '0%'}, {'filesystem': '/dev/vda1', 'mounted': '/tmp', 'percent': '22%'}, {'filesystem': '/dev/vdb', 'mounted': '/etc/group', 'percent': '2%'}, {'filesystem': 'tmpfs', 'mounted': '/proc/acpi', 'percent': '0%'}, {'filesystem': 'tmpfs', 'mounted': '/proc/scsi', 'percent': '0%'}, {'filesystem': 'tmpfs', 'mounted': '/sys/firmware', 'percent': '0%'}]

[cmsdmwmbot]

Can one of the admins verify this patch?

[todor-ivanov]

looking good.
@mapellidario just a note, for your consideration, not to request a change. This:

we show the value of the latest column "mounted on", which is not meaningful nor consistent across agents

Is because df command does not display duplicated mounts by default until explicitly told with the -a option. Which is the case for docker bind mounts starting from the same volume - in our case /dev/vdb. And the inconsistency between agents is only because df shows only the first mount, which basically at startup time succeeds on a random basis, so one never knows which of the so bound mounts would be visualized.

[mapellidario]

I do not want to use the -a option otherwise we would report that N disks are getting full when only one is indeed

(WMAgent-2.3.5) [cmst1@vocms0262:~]$ df -klPa | grep vdb
/dev/vdb         257837508 3999808 240714116       2% /etc/passwd
/dev/vdb         257837508 3999808 240714116       2% /etc/group
/dev/vdb         257837508 3999808 240714116       2% /data/certs
/dev/vdb         257837508 3999808 240714116       2% /data/admin/wmagent
/dev/vdb         257837508 3999808 240714116       2% /data/srv/wmagent/2.3.5/logs
/dev/vdb         257837508 3999808 240714116       2% /data/srv/wmagent/2.3.5/config
/dev/vdb         257837508 3999808 240714116       2% /data/srv/wmagent/2.3.5/install

[amaltaro]

Dario, these changes look good to me.
However, can you please enhance the relevant unit test to give us better coverage (more meaningful test):
https://github.com/dmwm/WMCore/blob/master/test/python/Utils_t/Utilities_t.py#L167
?

[mapellidario]

Changing the unit tests would mean parsing some output that is specific to the platform that is running the unittest (linux host, vm, container, osx, ...) and it is not easy to come up with some common pattern that is both meaningful and does not break on some machines, which i think was the reason behind that very generic unittest. I will give it a try, but i do not promise anything

[mapellidario]

So, Utils.Utilities.diskUse() in a jenkins node that runs our tests return the following output [1]

---

[1]

[
  {
    "filesystem": "devtmpfs",
    "mounted": "/dev",
    "percent": "0%"
  },
  {
    "filesystem": "tmpfs",
    "mounted": "/dev/shm",
    "percent": "0%"
  },
  {
    "filesystem": "tmpfs",
    "mounted": "/run",
    "percent": "11%"
  },
  {
    "filesystem": "tmpfs",
    "mounted": "/sys/fs/cgroup",
    "percent": "0%"
  },
  {
    "filesystem": "/dev/vda1",
    "mounted": "/",
    "percent": "76%"
  },
  {
    "filesystem": "/dev/vda15",
    "mounted": "/boot/efi",
    "percent": "3%"
  },
  {
    "filesystem": "cvmfs2",
    "mounted": "/cvmfs/cvmfs-config.cern.ch",
    "percent": "82%"
  },
  {
    "filesystem": "cvmfs2",
    "mounted": "/cvmfs/cms.cern.ch",
    "percent": "82%"
  },
  {
    "filesystem": "cvmfs2",
    "mounted": "/cvmfs/cms-ib.cern.ch",
    "percent": "82%"
  },
  {
    "filesystem": "cvmfs2",
    "mounted": "/cvmfs/grid.cern.ch",
    "percent": "82%"
  },
  {
    "filesystem": "cvmfs2",
    "mounted": "/cvmfs/projects.cern.ch",
    "percent": "82%"
  },
  {
    "filesystem": "cvmfs2",
    "mounted": "/cvmfs/unpacked.cern.ch",
    "percent": "82%"
  },
  {
    "filesystem": "tmpfs",
    "mounted": "/run/user/501",
    "percent": "0%"
  }
]

[mapellidario]

I added a new set of checks in that unit test, i am satisfied :)

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 1 new failures
  • 2 changes in unstable tests
• Python3 Pylint check: succeeded
  • 28 comments to review
• Pylint py3k check: failed
  • 4 errors and warnings that should be fixed
• Pycodestyle check: succeeded
  • 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15294/artifact/artifacts/PullRequestReport.html

[amaltaro]

It looks good to me, thanks Dario.

NOTE: with this change, we will have to push new couchapps in in the central CouchDB instance(s) when an upgrade time comes.