Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metrics to collect during training #3

Closed
wanchaol opened this issue Jan 23, 2024 · 4 comments · Fixed by #56, #57, #60 or #151
Closed

Add metrics to collect during training #3

wanchaol opened this issue Jan 23, 2024 · 4 comments · Fixed by #56, #57, #60 or #151
Milestone
First OSS release...

Comments

@wanchaol
Copy link
Contributor

see https://github.com/pytorch-labs/torchtrain/blob/main/train.py#L87

we should have the following metrics associated with the train steps:

  1. gpu memory usage
  2. wps
  3. loss
@wanchaol wanchaol changed the title Add metrics to collective during training Add metrics to collect during training Jan 23, 2024
@lessw2020
Copy link
Contributor

I can work on this one, with gpu stats first.
I think we also want mfu.

@lessw2020
Copy link
Contributor

and also flop counter

This was linked to pull requests Feb 16, 2024
add model num params display, gpu memory metrics #56
Merged
add TensorBoard logging with loss and wps #57
Merged
add memory metrics to TensorBoard #60
Merged
@awgu awgu mentioned this issue Mar 5, 2024
Closed
@tianyu-l tianyu-l linked a pull request Mar 19, 2024 that will close this issue
add MFU to metrics #151
Merged
@gnadathur
Copy link
Contributor

@lessw2020 , @tianyu-l -- Can we close this issue ?

@gnadathur
Copy link
Contributor

closing as per conversation w/ @lessw2020

jinsun-yoo pushed a commit to jinsun-yoo/torchtitan that referenced this issue Oct 30, 2024
@tianyu-l
adapt to the memory fix in pytorch (pytorch#3)
c0b08ed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
First OSS release of Torchtrain
3 participants
@gnadathur @wanchaol @lessw2020