Need Accurate Job Status, Failure Reason (Category), and useful Resolutions for users to take action

[scarlett2018]

Work break down:

• PAI Job Error Spec (Error Type, Error Code, Error Action, etc.) Design [Design] PAI Job Error Spec (Error Type, Error Code, Error Action, etc.) #2399
• PAI Job Error Spec, Collection and Handling [Backend]: PAI Job Error Spec, Collection and Handling - Phase II #2483 @yqwang-ms
  • Spec Table PAI Job Error: Spec Table #2485: @yqwang-ms
    1. Setup Initial Spec Config file
  • Launcher PAI Job Error: Launcher Enhancement #2484: @yqwang-ms
    1. Preserved Known User Container Exit Code
    2. Preserved Unknown User Container Exit Code
    3. Support structured ExitDiagnostics
  • YARN PAI Job Error: YARN Enhancement #2486: @yqwang-ms
    1. YARN Collect Runtime Dynamic ExitInfo (Runtime.pai.agg.error)
  • Runtime PAI Job Error: Runtime Enhancement #2487: @mzmssg @xudifsd @Binyang2014
    1. Refine ExitCode
    2. Split platform log from user log
    3. Generate dynamic ExitInfo into Runtime.pai.agg.error file
  • DiskCleaner PAI Job Error: DiskCleaner Enhancement #2488: @wangdian
    1. Print User Understandable Container Disk Error Message to DiskCleaner.pai.error file under Container log directory
  • RestServer PAI Job Error: RestServer Enhancement #2489: @Gerhut @sunqinzheng
    1. Setup and Load Spec
    2. Get more structured ExitInfo from static ExitSpec and dynamic ExitDiagnostics
  • WebPortal PAI Job Error: WebPortal Enhancement #2490 Add better error message for Exit code 177 for Job details page #2456: @Gerhut @sunqinzheng
    1. Show more structured ExitInfo
    2. Add Error Message and Retry info in UI
• Event sevice to provide more detailed failure reasons
• Refine status based on experiences (test case) to be itemlized

=============
Need Accurate Job Status, Failure Reason (Category), and useful Resolutions for users to take action

previous problems are also listed:

• need to know when the job is killed by system (i.e. preempt bonus).
• provide detailed reasons for "waiting" jobs  #824
• How to use GPUs on different machines? And how to change maxGPUs? #1832
• failed during to run job #1819
• [job status] Task status shows running, but no resources are allocated.  #2022
• Provide a better failure message experience for GPU resource request failure #2118

Job Status

• Job Submitted
  info needed:
  • job start time
  • retry times
• Job Waiting
  info needed:
  • sub task status: Task queueing; Task allocating resource; Task waiting;
  • job waiting time
• Job Running
  info needed:
  • sub task status: Task waiting; Task running; Task Completed; Task failed; Task Stopped;
  • job execution duration
• Job Stopped
  info needed:
  • job stopped at (time)
  • job stopped by (system/user)
  • job stopped reason
• Job Succeeded
  info needed:
  • job end time
• Job Failed
  info needed:
  • job failed at (time)
  • Error Type
  • Error Code
  • Error Message
  • Matched Pattern/resolution

Error Type (this need to be discuss and update after team discussion)

• User Command
• Launcher preparation failure
• Docker failure
• AM and other type (i.e. Error code 177)

Matched Pattern
Apply user code analyze rules for user command sub type error.

##Test Cases
Test Case 1. An example for resouce not enough "waiting":
“GangAllocation cannot be satisfied in time: Still waiting for 1 outstanding Tasks after timeout 1100s, maybe current available resource for the application is not enough, please retry later.”

Test Case 2. #2118
http://*.225:9286/view.html?username=Guisu&jobName=bert-pytorch-corn-18

The job requested 6 GPUs, but the server actually only have 4 as Max.
image
In the application summary log: http://*.225:9286/view.html?username=Scarlett&jobName=scarlett-bert-pytorch-corn-18
[ExitCustomizedDiagnostics]:
onError called into AM from RM due to non-transient error, maybe application is non-compliant.
Exception:
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested GPUs < 0, or requested GPUs > max configured, requestedGPUs=6, maxGPUs=4

Test Case 3 need to know when the job is killed by system #2333
NNI reports there are some failure jobs, but the status is not clear enough to let them know whether the job is killed by system or failed due to user codes.

[scarlett2018]

An example for resouce not enough "waiting":
“GangAllocation cannot be satisfied in time: Still waiting for 1 outstanding Tasks after timeout 1100s, maybe current available resource for the application is not enough, please retry later.”

[sterowang]

How about we display a timeline graph to show this curves together with job status changing?

1. How many containers has been allocated.
2. How many containers pending.
3. How many containers running.
4. How many containers succeeded.
5. How many containers failed.
6. How many containers stopped.

[yqwang-ms]

This PR covers latest per container state:
https://github.com/Microsoft/pai/pull/2306/files

For timeline graph, we may need to store status history first.