Python SDK and CLI End-Jun release plan

[mydmdm]

Add the PR of #3018

For the preview release, we will provide a basic functions of the command line tools and python SDK for below scenarios

1. Command line

• User can specify which cluster to interact with

opai cluster add -a <cluster-alias> --pai-uri http://x.x.x.x --user myuser  && opai cluster select <cluster-alias> 
opai cluster list
opai cluster select <cluster-alias>
opai delete

• User can submit and query jobs

opai job list -a <cluster-alias> <job-name> [config/ssh/stdout/stderr]
opai job submit -a <cluster-alias> <job-config-file> (json or yaml)

• ? how to get stdout / stderr by rest api or a link?
• when submit, user could change key-value from config file

opai job submit --update defaults:virtualCluster=test <job-config-file> 

• User can submit a simple job with shortcut

opai job sub -i <docker-image> -j <job-name> [--gpu 1] python a/b/c.py arg1 arg2 …

• User can access storage (only support hdfs for this release)

opai storage upload local_path remote_path
opai storage download remote_path local_path
opai storage list/status/delete remote_path

• User can leverage the jupyter notebook by command line

opai job notebook -i <docker-image> -w <workspace> -a <cluster-alias> <notebook>

this command will

• upload the <notebook> to the storage in <cluster> and submit a job
• execute it in the job container by jupyter nbconvert --execute and upload result (html) back
• wait job completed, and download html file

User may also add --interactive model to start and interact the server remotely

• parse ip address and states information from logging streams
• open browser automatically

2. python binding (SDK)

2.1 cluster management

• User can describe a cluster with openpaisdk.core.ClusterList class to describe multiple clusters

clusters = ClusterList().load() # defaultly loaded from "~/.openpai/clusters.json"

User add, delete methods to update clusters, select and get_client methods to select one from multiple clusters

• the Cluster class has methods to query and submit jobs

client = clusters.get_client(alias)
client.jobs(name)
client.rest_api_submit(job_config)

• the Cluster class has methods to access storage (through WebHDFS only for this version)

Cluster(...).storage.upload/download(...)

2.2 Job management

• User can describe a job with openpaisdk.core.Job class, which is compatible with the v2 protocol

job = Job(name)
job.submit(cluster_alias) # submit current job to a cluster

• provide some quick template of simple jobs

job.one_liner(...) # generate job config from a command
job.from_notebook(...) # turn notebook to job, automatically get current notebook name

[mydmdm]

#3018