2020 July ~ Aug Release

[scarlett2018]

This plan captures the work in July 2020. This is a 6 week iteration major focus on system features and improvements, and we will ship it in the beginning of August.

Release Manager

@yiyione

Endgame

Code freeze: 8.24
Scrum demo date: 8.25
Endgame & retrospective date: 8.31

Plan Items

Below is a summary of the top level plan items.

1. DB migration New RestServer Architecture: RestServer -> DB -> ApiServer #4651 TestOwner: @debuggy
   New RestServer Arch: RestServer -> DB -> ApiServer

• P0 - @hzy46 - job submit, view job PR: New RestServer Architecture: RestServer -> DB -> ApiServer; P0 Items #4761
• P0 - @yangou1988 refactor the current table columns (rename "Date modified" @hzy46 New RestServer Architecture: RestServer -> DB -> ApiServer; P0 Items #4761),
• P1 - Enable the ability to customize column by user.
• P0 - @yiyione - webportal paging
• P1 - Guoxin - Framework history
• P0 - Tests Database backend - stress test and failure test plan #4818

2. User and Group Management TestOwner: @fanyangCS

• P0 - @yiyione - VC && Group experience for Admin/User

3. Error message for SKU (i.e. CPU only job errors) TestOwner: @debuggy

• P0 - @abuccts - some of the error messages for resource constraints are miss leading or not actionable to users Wrong resource usage in UI #4602

4. Robustness improvement features User job write too much logs will cause disk pressure #4694 TestOwner: @abuccts

• P0 - @Binyang2014 - PAI service POD priority; disk pressure option;

5. HiveD improvement (aware bad nodes), bug fixes, and demo - @abuccts TestOwner: @yiyione

• P0 bug fixes code merged, need test
• P0 feature/hackathon demo

6. Marketplace @debuggy / @TobeyQin TestOwner: @TobeyQin

• P0 - Mingliang - unified PAI protocol Utilize job protocol in market item openpaimarketplace#49
• P0 - Mingliang - merge examples, merge marketplace (as part of the quick start deployment)
• P1 - @TobeyQin - new features (items)

9. AKS engine - @abuccts

• P0 shorten and simplify the Azure OpenPAI deployment
• P0 Quick start for Azure, and deploy CNI issue @abuccts

10. PAI DRI & Wiki

• Use PAI DRI notebook, DRI tickets and Wiki to update document @hzy46 Use PAI DRI notebook, DRI tickets, Wiki to update document #4760 broken link of pai wiki maintain machine & service #1763 PR: Fix document using DRI tickets #4828 Add distributed training examples of PyTorch #4821

11. Webportal api/v1 code update TestOwner: @debuggy

• webportal api/v1 code update webportal api/v1 and yarn fallback test #4704 @yiyione

12. Runtime image TestOwner: @hzy46

• Runtime image check issue Image check doesn't handle 429 response correctly openpai-runtime#17 @Binyang2014

stretch goal

7. Hybrid or multi-cluster management (SHAIIC)

• P0 Design review and MVP @mydmdm @scarlett2018
• P0 hackathon demo

8. P1 Elastic DL and Job level scale up and scale down - @yqwang-ms

• check with Ming/NNI and other integration scenarios
• TODO: summarize the to-the-customer level scenarios supports

9. rest server body size limit

• P0 [Rest Server] 100K body size limit in post request [Rest Server] 100K body size limit in post request #4750 (SHAIIC @mydmdm )

10. P2 REST server history and detail page code refactor @debuggy add new eslint rules and fix errors #4823 TestOwner: @yiyione

• code duplication, might need code refactor
• TODO: @debuggy discuss with @yqwang-ms
• Also show "More Diagnostics" in job retry history page Also show "More Diagnostics" in job retry history page #4689 @debuggy
• Move "More Diagnostics" button to job level and disable its scroll button Move "More Diagnostics" button to job level and disable its scroll button #4676 @debuggy

11. P1 Utilization support new features @hzy46 @Binyang2014

• Alert manager based utilization enhancement @suiguoxin
• GPU Utilization improve training cost ROI for EDU scenario @hzy46 Use alert-manager to alert/kill low GPU efficiency job #4623
• 🔴 GPU utilization for EDU scenario @TobeyQin @hzy46

12. P1 Cloud autoscale - @abuccts - scale up and scale down

• TODO: investigate whether official autoscale is fit for OpenPAI

13. P1 (depends on DB migration and refactor item) Surfacing more backend error to users in Job Details Page (@debuggy ,@yqwang-ms) Enrich job debugging info #4649

• @yqwang-ms listed of the top potential error in OneNote
• Show "More Diagnostics" in job detail page Add more task debug info in jobInfo api response [rest-server] #4667 Add task debug info in job detail page [webportal] #4670 @debuggy TestOwner: @yqwang-ms TestDone
• Show events in job detail and history page Enrich job debugging info #4649 @debuggy
• list out the potential scenario and possible display experience (i.e. advanced diagnostic mode) @debuggy

17. rotate-log issue @Binyang2014

• size rotate
• webportal get last two log file merge

18. P1 Release process speed up @yiyione
19. The defaulting should only be done after submission phase The defaulting should only be done after submission phase #4576

• P1 (defer candidate) @debuggy

20. P1 a image with Isomorphic supports for both CPU and GPU debug/deploy (SHAIIC)

21. P1 DShuttle Dshuttle integration Plan #4599

• Webportal support DShuttle type @yiyione Webportal use new storage API and support DShuttle type #4605
• DShuttle integration and internal deploy & usage @Binyang2014 (depending on the above item; ) Dshuttle integration Plan #4599
• TODO: @Binyang2014 breakdown and share a small set of release plan (book an end 2 end user experience review meeting)

22. Following up items of DB

• P1 archive (delete) jobs (SHAIIC) Support delete/archive jobs #4453
• P2 Job tags and favorite jobs (SHAIIC)
• [comments] DB replication; Master/Slave; SQL Azure; Recover method; P2 //(pending)job GC

---

TODO: all - review owned issues and put P0 and P1 into current plan

• open bugs: bug
  --- When job/task not completed, its completion time on WebPortal and RestServer is 0 (1970/1/1), confusing When job/task not completed, its completion time on WebPortal and RestServer is 0 (1970/1/1), confusing #4711

[scarlett2018]

Postpone items had been moved to $4512.