Need to know all the jobs in a Node

[scarlett2018]

One useful scenario is: Before Node upgrade/migration/deletion, admin need to know all the jobs in the corresponding node. there are more other scenarios.

[fanyangCS]

close and track in #349

[scarlett2018]

@fanyangCS - I'm reopening the issue as it is a display and query ask for Node<>job list, not just for decommission. I rephrased the job accordingly.

[xudifsd]

related PR #2197

[xudifsd]

#2197 merged. But seems user want something like, jobs with low resource util are marked with red:

We can add extra column to following page:

Also user want ordinary user could see the page instead of admin.

Should we discuss this? @scarlett2018 and @fanyangCS @Gerhut @Anbang-Hu

[scarlett2018]

@xudifsd - Agree the services page seems the most fit place for this request for now.

[xudifsd]

Webportal can query prometheus using query task_gpu_percent to get node, job mapping and display correctly in the page, the result of query should be like task_gpu_percent{container_env_PAI_TASK_INDEX="0",container_label_PAI_CURRENT_TASK_ROLE_NAME="worker",container_label_PAI_HOSTNAME="paigcr-a-gpu-1104",container_label_PAI_JOB_NAME="admin~dixu-gpu-test",container_label_PAI_USER_NAME="admin",instance="10.151.40.4:9102",job="pai_serivce_exporter",minor_number="0",pai_service_name="job-exporter",scraped_from="job-exporter-wxhc4"}, instance is where this job exist, and other labels can be extracted to make up job name.

For low resource usage, Webportal can query using task_gpu_percent < 50 and task_gpu_mem_percent < 50 to get low resource util of job, assume we only care gpu, and threshold for low is 50%.

[scarlett2018]

#2197 merged. But seems user want something like, jobs with low resource util are marked with red:

We can add extra column to following page:

Also user want ordinary user could see the page instead of admin.

Should we discuss this? @scarlett2018 and @fanyangCS @Gerhut @Anbang-Hu

Created an issue for this ask in #2250

[xudifsd]

close this and discuss in #2250