README and benchmark improvements #867
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: quantization README: 1) added fp6 to benchmarks 2) rewrote autoquant section to give a higher level explanation before diving into the details 3) reordered affine quantization section to first show techniques then dive into details 4) added fp6 section 5) moved kv cache stuff to new section 6) added sparse-marlin section and removed sparse-marlin benchmark from top of README since we don't have a reasonable flow for users to use to apply it to their model without a pre-sparsified checkpoint. 7) added uintx section Benchmarks Changes: 1) added instructions for adding things to benchmarks so everything stays consistent (in llama benchmark README) 2) organized/ran benchmarks for uintx and fp6 and sparse-marlin 3) added evaluations.sh to mirror benchmarks.sh 4) added sparse-marlin to eval.py 5) fixed some generate.py logging bugs 6) improved generate help quantization help text 7) fixed some eval.py bugs with uintx 8) added marlin to eval 9) fixed eval help text sparsity readme: 1) added some details to sparsity Test Plan: benchmarks.sh evaluations.sh Reviewers: Subscribers: Tasks: Tags:
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/867
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit e6db619 with merge base e283743 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
jerryzh168
reviewed
Sep 10, 2024
jerryzh168
approved these changes
Sep 10, 2024
jerryzh168
reviewed
Sep 10, 2024
| @@ -1,28 +1,23 @@ | |||
| README BENCHMARKS | |||
There was a problem hiding this comment.
btw, is this file manually organized?
There was a problem hiding this comment.
kind of, i organized it manually based on where the stuff would fall if you ran the entire benchmark suite, but since i ran them in different batches, they didn't come out all in the right order. And i added a few comments to make it easier to parse
jerryzh168
reviewed
Sep 10, 2024
| @@ -1,44 +1,28 @@ | |||
| export CHECKPOINT_PATH=../../../checkpoints # path to checkpoints folder | |||
There was a problem hiding this comment.
I think one improvement for this script might be to use a loop to only specify quantization or whatever arg that changes, instead of spelling out all the args, it might be easier to see what benchmarks we are running
There was a problem hiding this comment.
that would be good if we were productionizing things but most of the time i'm commenting things out and only running 1 or 2 examples and having them all spelt out is really nice
jainapurva
pushed a commit
that referenced
this pull request
Sep 22, 2024
README improvements Summary: quantization README: 1) added fp6 to benchmarks 2) rewrote autoquant section to give a higher level explanation before diving into the details 3) reordered affine quantization section to first show techniques then dive into details 4) added fp6 section 5) moved kv cache stuff to new section 6) added sparse-marlin section and removed sparse-marlin benchmark from top of README since we don't have a reasonable flow for users to use to apply it to their model without a pre-sparsified checkpoint. 7) added uintx section Benchmarks Changes: 1) added instructions for adding things to benchmarks so everything stays consistent (in llama benchmark README) 2) organized/ran benchmarks for uintx and fp6 and sparse-marlin 3) added evaluations.sh to mirror benchmarks.sh 4) added sparse-marlin to eval.py 5) fixed some generate.py logging bugs 6) improved generate help quantization help text 7) fixed some eval.py bugs with uintx 8) added marlin to eval 9) fixed eval help text sparsity readme: 1) added some details to sparsity Test Plan: benchmarks.sh evaluations.sh Reviewers: Subscribers: Tasks: Tags:
jainapurva
pushed a commit
that referenced
this pull request
Sep 23, 2024
README improvements Summary: quantization README: 1) added fp6 to benchmarks 2) rewrote autoquant section to give a higher level explanation before diving into the details 3) reordered affine quantization section to first show techniques then dive into details 4) added fp6 section 5) moved kv cache stuff to new section 6) added sparse-marlin section and removed sparse-marlin benchmark from top of README since we don't have a reasonable flow for users to use to apply it to their model without a pre-sparsified checkpoint. 7) added uintx section Benchmarks Changes: 1) added instructions for adding things to benchmarks so everything stays consistent (in llama benchmark README) 2) organized/ran benchmarks for uintx and fp6 and sparse-marlin 3) added evaluations.sh to mirror benchmarks.sh 4) added sparse-marlin to eval.py 5) fixed some generate.py logging bugs 6) improved generate help quantization help text 7) fixed some eval.py bugs with uintx 8) added marlin to eval 9) fixed eval help text sparsity readme: 1) added some details to sparsity Test Plan: benchmarks.sh evaluations.sh Reviewers: Subscribers: Tasks: Tags:
yanbing-j
pushed a commit
to yanbing-j/ao
that referenced
this pull request
Dec 9, 2024
yanbing-j
pushed a commit
to yanbing-j/ao
that referenced
this pull request
Dec 9, 2024
…h#862) * Updating torch nightly to pick up aoti improvements in 128339 * Update the torch version to 2.5 * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867)
yanbing-j
pushed a commit
to yanbing-j/ao
that referenced
this pull request
Dec 9, 2024
* Initial Creation of a quantization directory * Moving qops * updating import * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Update Quant call using llama.cpp (pytorch#868) llama.cpp did a BC breaking refactor: ggerganov/llama.cpp@1c641e6 resulting in some of our CI breaking This updates our CI to match llama.cpp's schema * Updating torch nightly to pick up aoti improvements in 128339 (pytorch#862) * Updating torch nightly to pick up aoti improvements in 128339 * Update the torch version to 2.5 * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867)
yanbing-j
pushed a commit
to yanbing-j/ao
that referenced
this pull request
Dec 9, 2024
* Removing all references to HQQ * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Update Quant call using llama.cpp (pytorch#868) llama.cpp did a BC breaking refactor: ggerganov/llama.cpp@1c641e6 resulting in some of our CI breaking This updates our CI to match llama.cpp's schema * Updating torch nightly to pick up aoti improvements in 128339 (pytorch#862) * Updating torch nightly to pick up aoti improvements in 128339 * Update the torch version to 2.5 * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Creating an initial Quantization Directory (pytorch#863) * Initial Creation of a quantization directory * Moving qops * updating import * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Update Quant call using llama.cpp (pytorch#868) llama.cpp did a BC breaking refactor: ggerganov/llama.cpp@1c641e6 resulting in some of our CI breaking This updates our CI to match llama.cpp's schema * Updating torch nightly to pick up aoti improvements in 128339 (pytorch#862) * Updating torch nightly to pick up aoti improvements in 128339 * Update the torch version to 2.5 * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867)
yanbing-j
pushed a commit
to yanbing-j/ao
that referenced
this pull request
Dec 9, 2024
* Removing GPTQ from all of torchchat * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Rebase + Add back accidental deletion * Update Quant call using llama.cpp (pytorch#868) llama.cpp did a BC breaking refactor: ggerganov/llama.cpp@1c641e6 resulting in some of our CI breaking This updates our CI to match llama.cpp's schema * Updating torch nightly to pick up aoti improvements in 128339 (pytorch#862) * Updating torch nightly to pick up aoti improvements in 128339 * Update the torch version to 2.5 * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Creating an initial Quantization Directory (pytorch#863) * Initial Creation of a quantization directory * Moving qops * updating import * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Update Quant call using llama.cpp (pytorch#868) llama.cpp did a BC breaking refactor: ggerganov/llama.cpp@1c641e6 resulting in some of our CI breaking This updates our CI to match llama.cpp's schema * Updating torch nightly to pick up aoti improvements in 128339 (pytorch#862) * Updating torch nightly to pick up aoti improvements in 128339 * Update the torch version to 2.5 * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Removing all references to HQQ (pytorch#869) * Removing all references to HQQ * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Update Quant call using llama.cpp (pytorch#868) llama.cpp did a BC breaking refactor: ggerganov/llama.cpp@1c641e6 resulting in some of our CI breaking This updates our CI to match llama.cpp's schema * Updating torch nightly to pick up aoti improvements in 128339 (pytorch#862) * Updating torch nightly to pick up aoti improvements in 128339 * Update the torch version to 2.5 * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Creating an initial Quantization Directory (pytorch#863) * Initial Creation of a quantization directory * Moving qops * updating import * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867) * Update Quant call using llama.cpp (pytorch#868) llama.cpp did a BC breaking refactor: ggerganov/llama.cpp@1c641e6 resulting in some of our CI breaking This updates our CI to match llama.cpp's schema * Updating torch nightly to pick up aoti improvements in 128339 (pytorch#862) * Updating torch nightly to pick up aoti improvements in 128339 * Update the torch version to 2.5 * Updating lm_eval version (pytorch#865) Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets * Pinning numpy to under 2.0 (pytorch#867)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
quantization README:
diving into the details
dive into details
top of README since we don't have a reasonable flow for users to use
to apply it to their model without a pre-sparsified checkpoint.
Benchmarks Changes:
stays consistent (in llama benchmark README)
sparsity readme:
Test Plan:
benchmarks.sh
evaluations.sh
Reviewers:
Subscribers:
Tasks:
Tags: