v0.14.0: End June 2019 Release

[debuggy]

Release v0.14.0

New Features

• Web portal: feature owner: @debuggy tester owner: @Dian @debuggy @ZhiyuanHe

  • New job submission page for pai protocol v2 New job submission page for pai protocol v2 #3026
  • Update home page to support dedicated vc [Web Portal] Update home page to support dedicated vc #2995

• Python SDK feature owner: @mydmdm ; tester owner: @sunqinzheng

  • Sdk release v0.4.00 Sdk release v0.4.00 #3018

• New scheduler: feature owner: @mzmssg ; tester owner: @abuccts

  • Dedicated vc support Dedicated vc support #2960

• PAI vscode extension: feature owner: @yiyione ; tester owner: @sunqinzheng

  • Submit job v2 support [VS Code] Submit job v2 support #2913
  • Add schema and snippets for YAML text editor. [VS Code] Add schema and snippets for YAML text editor. #2978

• Team storage plugin: feature owner: @dianwang ; tester owner: @debuggy @Binyang2014

  • New team-wise manage cli New team-wise manage cli #2943

Improvements

• Web portal:

  • Refine job detail page's task list [Web Portal] Refine job detail page's task list #2953
  • add new webHDFSUri in env.js.template ( fix bug when pylonAddress not set in .env  #3048)
  • Tweak job submission page layout ([Web Portal] Tweak job submission page layout #3043)
  • css tweak ([Web Portal] job submission page css tweak #3041)
  • refine submit job UI page (refine submit job UI page #3037)
  • refine home page's error handling ([Web Portal] refine home page's error handling #3196)
  • renew docker image list and add tooltip (renew docker image list and add tooltip #3181)
  • remove prettier config file (remove prettier config file #3184)
  • remove tachyons css to avoid classname error (remove tachyons css to avoid classname error #3173)
  • Add confirm dialog before batch edit admin's password ([Web Portal] Add confirm dialog before batch edit admin's password #3174)
  • fix UI broken if choose all of the VC when create user ([Web Portal] fix UI broken if choose all of the VC when create user #3177)
  • redesign batch edit behavior ([Web Portal] Redesign batch edit behavior #3172)
  • trim the docker url after job submission; fix job detail page's clone button's padding ([Web Portal] trim the docker url after job submission; fix job detail page's clone button's padding #3169)
  • Change 'Import CSV' to 'Create Bulk Users' in user management ([Web Portal] Change 'Import CSV' to 'Create Bulk Users' in user management #3136)
  • update command section's placeholder ([Web Portal] update submision page's command section's placeholder #3150)
  • move documents link to top nav bar ([Web Portal] move documents link to top nav bar #3126)
  • tweak home page's gpu chart's height ([Web Portal] tweak home page's gpu chart's height #3131)
  • display red border when a task role is invalid ([Web Portal] display red border when a task role is invalid #3072)
  • change label of container size to resources per instance (change label of container size to resources per instance #3101)
  • add error message when command is empty (add error message when command is empty #3098)
  • disable edit user form's auto fill ([Web Portal] disable edit user form's auto fill #3107)

• Rest server: feature owner && tester owner: @abuccts @ydye @sunqinzheng

  • Add get job api v2 [Rest Server] Add get job api v2 #2851
  • Fix error in Docker runtime detection [Rest Server] Fix error in Docker runtime detection #2888
  • update user's vc permission when create/remove vc [Rest Server] update user's vc permission when create/remove vc #2939
  • Remove deprecated api in rest server Remove deprecated api in rest server #2954
  • AAD & New Schema of User and Group (AAD & New Schema of User and Group #3034)
  • change get vc api (Binyli/change get vc api #3149)

• Yarn cluster && Framework launcher: feature owner && tester owner: @mzmssg

  • Upgrade zookeeper version to 3.4.14 upgrade zookeeper to 3.4.14 #2884
  • Upgrade to Zookeeper 3.4.14 [Launcher]: Upgrade to Zookeeper 3.4.14 #2911

• Deployement: feature owner && tester owner: @debuggy

  • client tolerate datanode replacement failure [HDFS] client tolerate datanode replacement failure #2895
  • "paictl service delete" supports to delete log "paictl service delete" supports to delete log #2926
  • Update maximum resources in yarn-site.xml Update maximum resources in yarn-site.xml #2942
  • 418.56 support [Drivers] 418.56 support #3020
  • upgrade pai version in k8s (609680)
  • upgrade pai version (upgrade pai version #3056)
  • Remove duplicate version prefix ([Webportal] Remove duplicate version prefix #3124)

• Plugins: (no test)

  • Upgrade fstream version to fix vulnerability [Plugin] Upgrade fstream version to fix vulnerability #2886
  • Refine error message for marketplace [Plugin] Refine error message for marketplace #2998

• Security: (no test)

  • Update js-yaml Bump js-yaml from 3.12.0 to 3.13.1 in /src/webportal #2890 Bump js-yaml from 3.12.1 to 3.13.1 in /contrib/submit-simple-job #2891 Bump js-yaml from 3.12.2 to 3.13.1 in /contrib/submit-nfs-job #2892
  • Add dependabot config file Add dependabot config file #2893
  • Update handlebars Bump handlebars from 4.0.12 to 4.0.14 in /src/webportal #2899 Bump handlebars from 4.0.12 to 4.1.2 in /src/rest-server #2900
  • Update dependabot config file Update dependabot config file #2914
  • Update is-my-json-valid Bump is-my-json-valid from 2.16.1 to 2.20.0 in /src/webportal #2931
  • Update twisted Bump twisted from 18.9.0 to 19.2.1 in /src/yarn-exporter/src #2912

Documentation

• Rewording and some format fixes. refine documents #2927
• Chinese translation and placeholder. Chinese translation and placeholder. #2919
• add submit v2 job to readme [VS Code] Add submit V2 job to readme #3017)
• api doc update ([User & Group] restserver api doc update.  #3216)
• add release note (add release note #3204)
• pai upgrade doc (pai upgrade doc #3195)
• add job submission docs (add job submission docs #3183)
• change link of external project like python sdk (change link of external project like python sdk #3237)

Bug Fixes

• Web portal: @scarlett2018

  • hide 0 gpu nodes from Available Nodes Chart [Web Portal] hide 0 gpu nodes from Avaliable Nodes Chart #2915
  • fix job detail page's gpu attributes display bug. [Web Portal] fix job detail page's gpu attributes display bug. #3027
  • disable submit if command is empty (disable submit if command is empty #3080)
  • auto remove empty lines in command (Binyli/commands #3074)
  • keep mount config selection state （0af835)
  • fix the set-state warning after clone job ([Web Portal] fix the set-state warning after clone job #3070)
  • fix clone job bug (fix clone job bug #3068)
  • fix <p> tag and prop-types warning ([Web Portal] fix submit job page's <p> tag and prop-types warning #3067)
  • Return empty command if no teamwise mounts (4afd2e)
  • Adjust team-mount-list view (1b4190)
  • Align job submit page's submission section to task role ([Web Portal] Align job submit page's submission section to task role card's text field #3065)
  • Add tooltips to job submission page's field label. ([Web Portal] Add tooltips to job submission page's field label. #3046)
  • remove plugin in webportal config (remove plugin in webportal config #3063)
  • remove pylon address dependency ([Web Portal] remove pylon address dependency #3040)
  • fix export yaml bug (fix export yaml bug #3047)
  • fix webhdfs wrong request (fix webhdfs wrong request #3044)
  • customized docker image inputField may disappear (bug fix: customized docker image inputField may disappear if docker uri is empty #3033)
  • add dependency of joi for node server (add dependency of joi for node server #3031)
  • remove duplicated v of feedback (remove duplicated v of feedback #3230)
  • change webportal doc link (change webportal doc link #3229)
  • fix stdout/stderr's full log link bug when pylon is not used ([Web Portal] fix stdout/stderr's full log link bug when pylon is not used #3219)
  • change tutorial link of home page (change tutorial link of home page #3213)
  • fix bug of GPU available number (fix bug of GPU available number #3210)
  • hot fix for hdfs CORS problem (hot fix for hdfs CORS problem #3145)
  • fix docker bug Binyli/bug fix #3134
  • hot fix hdfscli proxy problem (hot fix hdfscli proxy problem #3130)
  • fix virtual cluster's default value after job clone ([Web Portal] fix virtual cluster's default value after job clone #3128)
  • refine hdfs check for robustness (refine hdfs check for robustness #3116)
  • change to lowercase letter for 'Completion Policy' (change to lowercase letter for 'Completion Policy' #3119)
  • fix data command error (fix data command error #3111)
  • fix job submission page's jobRetryCount and taskRetryCount field ([Web Portal] fix job submission page's jobRetryCount and taskRetryCount field #3112)
  • redict v2 job to default submission papge if plugin not installed (fix bug: redict v2 job to default submission papge if plugin not installed #3091)
  • fix docker (Binyli/docker #3097)
  • add empty key check to key-value list control ([Web Portal] add empty key check to key-value list control #3096)
  • change command section's default comments to placeholder ([Web Portal] change command section's default comments to placeholder #3095)
  • disable submit if command is empty (disable submit if command is empty #3080)
  • fix deployment field missing bug (fix deployment field missing bug #3238)
  • [Web Portal] fix port list bug ([Web Portal] fix port list bug #3240)

• Rest server: feature owner && tester owner: @abuccts @ydye @sunqinzheng

  • Add quotes for masked secrets field in protocol
  • Trap SIGTERM in entrypoint to avoid yarn container early stop Trap SIGTERM in entrypoint to avoid yarn container early stop #2947
  • fix bug fix bug #3009
  • Update http errors in get job v2 [Rest Server] Update http errors in get job v2 #3022
  • User Migrate Script Fix. (User Migrate Script Fox. #3090)
  • Fix issue in updateUserVirtualCluster of rest-server (565073)
  • Fix user migration issue (Fix user migration issue #3036)
  • api permission fix ([hotfix] api permission fix #3211)
  • Fix AAD group in dedicated vc create/delete ([Hotfix] Fix AAD group in dedicated vc create/delete #3143)
  • API to create vc and remove vc and do the same operation to group (New API to create vc and remove vc and do the same operation to group #3064)
  • Groupname schema ([Hot fix] Groupname schema #3099)

• Hadoop: feature owner && tester owner: @mzmssg

  • Increase HDFS client default replica to 3 Increase HDFS client default replica to 3  #2924
  • port conflict [hadoop-ai] port conflict #3012
  • Pre check vclist from yarn and remove the vc type group which is not in. (Pre check vclist from yarn and remove the vc type group which is not in. #3158)
  • fix and improvement ([runtime] fix and improvement #3093)

• Deployment:

  • fix configMigration srcipt bug(Mintao/fix migration #3220)
  • user migrate script issue fix ([hotfix] user migrate script issue fix  #3209)
  • Move npm version to dockerfile Move npm version to dockerfile #3109 ([Hotfix] Move npm version to dockerfile #3109 #3110)

Known Issues

• All lines in command will be concatenated by &&, so use # or \ in the command will cause bugs. This will be fixed in the future.
• Based on official doc, the different gpu driver versions may support different cuda versions. As our tests, current 384.111 gpu driver version does not support cuda10 image.

[debuggy]

the release commits are updated