Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize e2e runtime: reduce pytorch-plugin image download time #2691

Merged
merged 3 commits into from
Feb 23, 2023
Merged

Optimize e2e runtime: reduce pytorch-plugin image download time #2691

merged 3 commits into from
Feb 23, 2023

Conversation

wangyang0616
Copy link
Member

@wangyang0616 wangyang0616 commented Feb 22, 2023 •
edited
Loading

It takes too long for e2e to run the pytorch plugin use case, basically between 300s and 600s, as follows:

• [SLOW TEST] [458.017 seconds]
Pytorch Plugin E2E Test will run and complete finally
/home/runner/work/volcano/volcano/test/e2e/jobseq/pytorch_plugin.go:12
  • The image pytorch-mnist:v1beta1-45c5727 is changed to pytorch-mnist:v1beta1-9ee8fda, and the image size is reduced from 1.2 GB to 600 MB.
  • Unnecessary image download operations on the host node are reduced. Images can be directly loaded on the kind virtual node.
  • The running time of e2e test cases is reduced from 38 minutes to about 30 minutes.

@volcano-sh-bot volcano-sh-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Feb 22, 2023
@wangyang0616 wangyang0616 changed the title Load the image required by e2e to the kind cluster in advance to save… Optimize e2e runtime: reduce pytorch-plugin image download time Feb 22, 2023
@volcano-sh-bot volcano-sh-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 22, 2023
@wangyang0616
Copy link
Member Author

/retest

@wangyang0616
The e2e test uses the pytorch-mnist image dumped from the volcanosh i…
bf92f52 
…mage warehouse to improve the image download speed

Signed-off-by: wangyang <wangyang8126@gmail.com>
@wangyang0616
Use a smaller pytorch-mnist mirror
d79ecc6 
Signed-off-by: wangyang <wangyang8126@gmail.com>
@wangyang0616
Delete the invalid image download operation, and automatically downlo…
f1202e3 
…ad the required image through the virtual node in the kind

Signed-off-by: wangyang <wangyang8126@gmail.com>
@william-wang
Copy link
Member

It's great to replace to image to reduce the e2e time.

william-wang
william-wang approved these changes Feb 23, 2023
View reviewed changes
Copy link
Member

@william-wang william-wang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@volcano-sh-bot volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Feb 23, 2023
@volcano-sh-bot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: william-wang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@volcano-sh-bot volcano-sh-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 23, 2023
@volcano-sh-bot volcano-sh-bot merged commit 71dc523 into volcano-sh:master Feb 23, 2023
@wangyang0616 wangyang0616 mentioned this pull request Feb 23, 2023
Closed
@wangyang0616 wangyang0616 mentioned this pull request Jul 29, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@william-wang william-wang william-wang approved these changes

@k82cn k82cn Awaiting requested review from k82cn

@merryzhou merryzhou Awaiting requested review from merryzhou

Assignees

@william-wang william-wang

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wangyang0616 @william-wang @volcano-sh-bot