Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[sharktank] Add Perplexity pre-submit test #579

Merged
merged 29 commits into from
Nov 23, 2024
Merged

[sharktank] Add Perplexity pre-submit test #579

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
814b364
Add perplexity pre-submit tests
archana-ramalingam Nov 21, 2024
9e965a2
Fix log indentation
archana-ramalingam Nov 21, 2024
3089e61
Add better logging
archana-ramalingam Nov 21, 2024
c1f7acc
Merge branch 'main' into perplexity-pre-submit
archana-ramalingam Nov 21, 2024
8266b62
Change hip device for pre-submits
archana-ramalingam Nov 21, 2024
8ea8a24
Merge branch 'perplexity-pre-submit' of https://github.com/nod-ai/sha…
archana-ramalingam Nov 21, 2024
93b6a2b
Add is_mi300x marker to skip ppl tests
archana-ramalingam Nov 21, 2024
e88a3ea
Merge branch 'main' into perplexity-pre-submit
archana-ramalingam Nov 21, 2024
c5d788e
Rename workflow and job for consistency
archana-ramalingam Nov 22, 2024
cca5a14
Merge branch 'main' into perplexity-pre-submit
archana-ramalingam Nov 22, 2024
1b1cec7
Merge branch 'perplexity-pre-submit' of https://github.com/nod-ai/sha…
archana-ramalingam Nov 22, 2024
33d25e1
Add --bs and adjust mean perplexity
archana-ramalingam Nov 22, 2024
453b4d2
Merge branch 'main' into perplexity-pre-submit
archana-ramalingam Nov 22, 2024
71a520d
Merge branch 'main' into perplexity-pre-submit
archana-ramalingam Nov 22, 2024
a2d7ac9
Update workflow names to be consistent
archana-ramalingam Nov 22, 2024
03bbced
Use batch_size fixture
archana-ramalingam Nov 22, 2024
1335166
Test longrun tests
archana-ramalingam Nov 22, 2024
2c6b191
Merge branch 'perplexity-pre-submit' of https://github.com/nod-ai/sha…
archana-ramalingam Nov 22, 2024
6626fa1
Correct bs to batch_size
archana-ramalingam Nov 22, 2024
1038878
Merge main changes
archana-ramalingam Nov 22, 2024
33c7995
Add page_cache_size calc to torch ppl
archana-ramalingam Nov 22, 2024
850c775
Remove exceptions
archana-ramalingam Nov 22, 2024
51be7c1
Merge branch 'main' into perplexity-pre-submit
archana-ramalingam Nov 22, 2024
29aadd3
update test name
archana-ramalingam Nov 22, 2024
9886d6f
Merge branch 'perplexity-pre-submit' of https://github.com/nod-ai/sha…
archana-ramalingam Nov 22, 2024
56de4e0
Revert Test longrun tests
archana-ramalingam Nov 22, 2024
df33088
Merge branch 'main' into perplexity-pre-submit
archana-ramalingam Nov 22, 2024
5dcac71
Opt in large tests with --run-nightly-llama-tests flag
archana-ramalingam Nov 23, 2024
ec02ffb
Merge branch 'perplexity-pre-submit' of https://github.com/nod-ai/sha…
archana-ramalingam Nov 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci_eval_short.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,4 +74,4 @@ jobs:
iree-base-runtime

- name: Run perplexity test with vmfb
run: pytest -n 8 -v -s sharktank/tests/evaluate/perplexity_iree_test.py --run-quick-llama-test --bs=5 --iree-device='hip://6' --iree-hip-target=gfx942 --iree-hal-target-backends=rocm --llama3-8b-f16-model-path=/data/llama3.1/8b/llama8b_f16.irpa --llama3-8b-tokenizer-path=/data/llama3.1/8b/tokenizer_config.json
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason we are targeting only device hip://6? Should we have the runners available decide the device used?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, but the specific iree_device needs to be passed to the vmfbRunner. We might require another script/flag to determine a free device and pass that info dynamically. Let me know if it's worth looking into.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saienduri have we setup ways to target specific devices on runners outside hardcoding like this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we do. We can specify ROCR_VISIBLE devices so that runners initialize with only certain gpus to avoid conflicts. I can make sure the seperation is in place to avoid conflicts on Monday.

run: pytest -n 8 -v -s sharktank/tests/evaluate/perplexity_iree_test.py --bs=5 --iree-device='hip://6' --iree-hip-target=gfx942 --iree-hal-target-backends=rocm --llama3-8b-f16-model-path=/data/llama3.1/8b/llama8b_f16.irpa --llama3-8b-tokenizer-path=/data/llama3.1/8b/tokenizer_config.json
4 changes: 2 additions & 2 deletions sharktank/tests/evaluate/perplexity_iree_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@

is_mi300x = pytest.mark.skipif("config.getoption('iree_hip_target') != 'gfx942'")
skipif_run_quick_llama_test = pytest.mark.skipif(
archana-ramalingam marked this conversation as resolved.
Show resolved Hide resolved
'config.getoption("run-quick-llama-test") and not config.getoption("run-nightly-llama-tests")',
reason="Skipping large tests when --run-quick-llama-test is set",
'not config.getoption("run-nightly-llama-tests")',
reason="Run large tests if --run-nightly-llama-tests is passed",
)


Expand Down
Loading