August 2019 Release Plan

[hzy46]

This plan captures our work in end July and August. (Internal release. https://github.com/microsoft/pai/releases/tag/v0.15.0)

Endgame

• TBD Code freeze for the endgame
• TBD Endgame done

The endgame details for this iteration are tracked in ticket to be created.

Plan Items

Expect: k8s merge. May need more discussion.

• Conduct user interviews and collect feedbacks @scarlett2018 @rabbit008 @debuggy @hzy46 @yiyione @yangou1988
  • Design a questionnaire or interview template
  • Plan for improvements or new functions based on user feedbacks
    • Job submission docs update (homepage link and ssh) PR: update job submission docs #3347
• Improve job submission page @Binyang2014 @debuggy @sunqinzheng
  • Quick wizards or templates for users New Job Wizard #3025 (can be included in user interviews) PR: add wizard page when submitting job #3430
  • Enhance data storage functions @wangdian (can be included in user interviews) [webportal] Refine team-wise data section in job submission Page #3294 PR: Refine team share storage in Job submission page. #3416
• Design JS SDK to encapsulate current RESTful APIs @yiyione @hzy46 PR: JS SDK init version. #3414
  • Go through current APIs OpenPAI JS SDK #3401 @yiyione
• Python SDK and Jupyter extension add Jupyter Notebook extension for job submission #3289 @mydmdm @hzy46 PR: Add Notebook Extension #3418
• AAD related issues: mapping between AAD groups and VCs [AAD] Decouple PAI internal resource from external user/groups  #3275 @mzmssg
  • Configuration file
  • RESTful API (optional)
• End-to-End AAD test cases ([AAD] End-to-End test AAD  #2898) @ydye PR: [Doc] AAD end2end document #3362
• Existing portal issues @sunqinzheng
  • Homepage Improve Available GPU board #3045 PR: [Web Portal] Refine home page #3265
  • User filter Filter User: It's very hard to find a user when the user list is very long #2970 PR: [Web Portal] refine job list page's filters #3310
• RESTful API update @yiyione
  • Docs Compare Rest-server APIs with Docs #3497
  • Cluster info api @sunqinzheng [Rest Server] add cluster info api #3281 PR: [Rest Server] add cluster info api #3281
• TensorBoard integration @TongboZhang ([TensorBoard] Redesign TensorBoard Integration #3156, Expose Tensorboard URL in Job detailed Page #2095) PR: [TensorBoard] Integrate TensorBoard on the Job Submission page #3257
• Refine current home page @Binyang2014 Refine Home Page for Admin Experience #2404 add a dashboard in grafana to list all tasks in node #2197 PR: Binyli/home page #3411
• A confirm dialog for stop actions. Add confirm dialog for stop actions to avoid mis-operation. #2577 @fenghuajia PR: [webportal]add stop job confirm #3408
• Add a button in job-detail page to get merged stdout&stderr log(Add a button in job-detail page to get merged stdout&stderr log #3246) @Binyang2014
• VSCode minor issues (VS Code Client UI minor issues. #2706) @yiyione
• Attach commit number when PAI is deployed for testing automatically attach code commit number to the webportal #3320 @hzy46 @debuggy PR: Attach a commit ID for webportal #3374
• SSH-related issues @hzy46
  • Build SSH-enabled docker images as recommendations Build docker images with SSH enabled #3355 PR: Update default docker images #3381
  • Add more information when SSH is disabled Add more information when SSH is disabled #3356 PR: Update SSH info for better user experience #3389

Engineering Improvements

Bug fixes

• A Hadoop bug in regex to match docker image name (submit job execute failed #3202) @mzmssg
• Docker's data-root will lost on Azure Node restart(docker's data-root will lost on Azure Node restart #2286) @ydye

Candidate Items

• Mismatch between job attempt and metric(Mismatch between job attempt and metric #2938)
• Alert for abnormal jobs (including long running and low utilization jobs) Alert for abnormal jobs (including long running and low utilization jobs) #2629
• Highlight jobs with low GPU utilization(Highlight low resource utilization jobs on webportal #2249)
• Create initial password for new user (Generate a random initial password for new user #3133)
• Refactor eslint of webportal (refactor eslint of webportal #3235) @debuggy
• A page for team storage
• Refine hardware page to break down per GPU usage(Refine hardware page and break down per GPU usage #3348)

Deferred

• Embedded necessary properties in job configuration
  • Examples: 1. submission signature 2. submission purpose
• New Marketplace
• Data storage management UI
• AAD Web UI (optional)
• Add better PyTorch example with "shmMB" in its settings Add an example for pytorch #3357

[fanyangCS]

Do you plan to work on marketplace in this release?

[scarlett2018]

Do you plan to work on marketplace in this release?

not yet, just added it to the "pending items" and let's discuss it in next scrum meeting.

[fanyangCS]

TensorBoard integration? @TongboZhang

[joveryz]

TensorBoard integration? @TongboZhang

Yes, hope this feature can be included in August release.

[hzy46]

TensorBoard integration? @TongboZhang

Yes, hope this feature can be included in August release.

TensorBoard integration has been added to plan items. @fanyangCS @TongboZhang

[fanyangCS]

A candidate feature in this release. #2577 @scarlett2018 @hzy46

[hzy46]

A candidate feature in this release. #2577 @scarlett2018 @hzy46

Added to pending items for further discussion.

[fanyangCS]

#2404 another candidate feature @hzy46

[hzy46]

#2404 another candidate feature @hzy46

Added to pending items.