Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add timm and huggingface model suites support #2197

Closed
wants to merge 19 commits into from
Closed

Conversation

xuzhao9
Copy link
Contributor

@xuzhao9 xuzhao9 commented Mar 14, 2024

Dynamobench supports extra huggingface and timm models beyond the existing model set in TorchBench.
This PR will add support to those models as well, and they can be invoked with run.py or in the group_bench userbenchmarks.

Test plan:

TIMM model example:

$ python run.py convit_base -d cpu -t eval
Running eval method from convit_base on cpu in eager mode with input batch size 64 and precision fp32.
CPU Wall Time per batch: 4419.601 milliseconds
CPU Wall Time:       4419.601 milliseconds
Time to first batch:         2034.6840 ms
CPU Peak Memory:                0.6162 GB
$ python run.py convit_base -d cpu -t train
Running train method from convit_base on cpu in eager mode with input batch size 64 and precision fp32.
CPU Wall Time per batch: 17044.825 milliseconds
CPU Wall Time:       17044.825 milliseconds
Time to first batch:         1616.9790 ms
CPU Peak Memory:                7.3408 GB

Huggingface model example:

python run.py MBartForCausalLM -d cuda -t train
Running train method from MBartForCausalLM on cuda in eager mode with input batch size 4 and precision fp32.
GPU Time per batch:  839.994 milliseconds
CPU Wall Time per batch: 842.323 milliseconds
CPU Wall Time:       842.323 milliseconds
Time to first batch:         5390.2949 ms
GPU 0 Peak Memory:             19.7418 GB
CPU Peak Memory:                0.9121 GB

Fixes #2170

@bhack
Copy link
Contributor

bhack commented Mar 15, 2024

Can we add something interesting here to stress test:
pytorch/pytorch#121072
pytorch/pytorch#121324 (comment)

/cc @ezyang @albanD @vfdev-5 @nWEIdia @xuzhao9 @eqy @ptrblck

@facebook-github-bot
Copy link
Contributor

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot facebook-github-bot requested a deployment to docker-s3-upload March 15, 2024 21:46 — with GitHub Actions Abandoned
@facebook-github-bot facebook-github-bot requested a deployment to docker-s3-upload March 15, 2024 21:57 — with GitHub Actions Abandoned
@facebook-github-bot
Copy link
Contributor

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@HDCharles
Copy link
Contributor

looks good, so if i put in extended models, does it still run the normal torchbench models or no? it'd be nice to be able to run each group individually if not.

Copy link
Contributor

@HDCharles HDCharles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see note in comment

@xuzhao9
Copy link
Contributor Author

xuzhao9 commented Mar 19, 2024

@HDCharles Yes, it will run both models specified in models: and extended_models: sections. However, currently we will run them as individual models and there is no "grouping" in terms of results. We could add that in a follow-up PR.

@facebook-github-bot
Copy link
Contributor

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@xuzhao9 merged this pull request in 2196021.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support timm_* and hf_* models
4 participants