Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MinorFix] Add code to record memory usage #45

Merged
merged 1 commit into from
Jan 24, 2025
Merged

[MinorFix] Add code to record memory usage #45

merged 1 commit into from
Jan 24, 2025

Conversation

LiGuihong
Copy link

@LiGuihong LiGuihong commented Jan 13, 2025 •
edited by sarooshsh
Loading

This PR will change the training output from

EXP_NAME #Nodes Model_SIZE Seq_Len MBS GBS TP PP CP Tokens/Sec/GPU TFLOPs/s/GPU Memory Usage Time/iter
perf 1 70 2048 8 64 8 1 1 1107.185008 479.61176470588236 nan 14797.88823529412

to

EXP_NAME #Nodes Model_SIZE Seq_Len MBS GBS TP PP CP Tokens/Sec/GPU TFLOPs/s/GPU Memory Usage Time/iter
perf 1 70 2048 8 64 8 1 1 1107.185008 479.61176470588236 0.7856 14797.88823529412

Note change in memory usage column.

@wenchenvincent
Copy link
Collaborator

Could you add the output from the change in the PR description?

@LiGuihong
Copy link
Author

Output will change from

EXP_NAME #Nodes Model_SIZE Seq_Len MBS GBS TP PP CP Tokens/Sec/GPU TFLOPs/s/GPU Memory Usage Time/iter
perf 1 70 2048 8 64 8 1 1 1107.185008 479.61176470588236 nan 14797.88823529412

to

EXP_NAME #Nodes Model_SIZE Seq_Len MBS GBS TP PP CP Tokens/Sec/GPU TFLOPs/s/GPU Memory Usage Time/iter
perf 1 70 2048 8 64 8 1 1 1107.185008 479.61176470588236 0.7856 14797.88823529412

sarooshsh
sarooshsh approved these changes Jan 15, 2025
View reviewed changes
Copy link

@sarooshsh sarooshsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! 👍

@sarooshsh sarooshsh changed the title add code to record memory usage [MinorFix] Add code to record memory usage Jan 15, 2025
@LiGuihong LiGuihong merged commit e35b94b into rocm_dev Jan 24, 2025
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gurpreet-dhami gurpreet-dhami gurpreet-dhami approved these changes

@sarooshsh sarooshsh sarooshsh approved these changes

@lcskrishna lcskrishna Awaiting requested review from lcskrishna

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@LiGuihong @wenchenvincent @gurpreet-dhami @sarooshsh