-
Notifications
You must be signed in to change notification settings - Fork 280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add timm and huggingface model suites support #2197
Conversation
Can we add something interesting here to stress test: /cc @ezyang @albanD @vfdev-5 @nWEIdia @xuzhao9 @eqy @ptrblck |
@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
looks good, so if i put in extended models, does it still run the normal torchbench models or no? it'd be nice to be able to run each group individually if not. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see note in comment
@HDCharles Yes, it will run both models specified in |
@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Dynamobench supports extra huggingface and timm models beyond the existing model set in TorchBench.
This PR will add support to those models as well, and they can be invoked with
run.py
or in thegroup_bench
userbenchmarks.Test plan:
TIMM model example:
Huggingface model example:
Fixes #2170