fix llama readme #3339
Closed
fix llama readme #3339
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…orch release/2.3 (#2818)
Summary: Pull Request resolved: #2919 The note directive on sphinx doesn't render well on markdown. Remove it to avoid cause confusion. Reviewed By: mergennachin, cccclai Differential Revision: D55881939 fbshipit-source-id: a4252f0b70593ecd97e5cc352c601e772a9c222a (cherry picked from commit dc7e4d5) Co-authored-by: Hansong Zhang <hsz@meta.com>
…2911) Summary: Update the LLM getting started guide for uniform tone and tense. Informally following the Google developer documentation style guide: https://developers.google.com/style. Also, resolve a number of outstanding issues with incorrect or misleading documentation and steps. For reference, here are links to the current and proposed LLM guide: https://docs-preview.pytorch.org/pytorch/executorch/2911/llm/getting-started.html (proposed) https://pytorch.org/executorch/main/llm/getting-started.html (live) Pull Request resolved: #2911 Reviewed By: Gasoonjia, byjlw Differential Revision: D55867181 Pulled By: GregoryComer fbshipit-source-id: 5e865eaa4a0ae52845963b15c221a3d272431448 (cherry picked from commit 01bac3d)
Summary: Pull Request resolved: #2926 Fixing issues we've seen in #2907 and #2805 bypass-github-export-checks bypass-github-pytorch-ci-checks bypass-github-executorch-ci-checks Reviewed By: iseeyuan, cccclai Differential Revision: D55893925 fbshipit-source-id: c6e0264d868cb487faf02f95ff1bd223cbcc97ac (cherry picked from commit 6db9d72)
Summary: Pull Request resolved: #2927 ATT Created from CodeHub with https://fburl.com/edit-in-codehub Reviewed By: mergennachin Differential Revision: D55895703 fbshipit-source-id: 5466b44224b8ebf7b88d846354683da0c1f6a801 (cherry picked from commit ce447dc)
Summary: Pull Request resolved: #2876 Fixing the tag constant for mutable buffer. The buffer shouldn't be tagged if it's going to be mutated by the delegated. It's more common in hardware backends Will follow up and test having delegate consume mutation Reviewed By: mcr229, angelayi Differential Revision: D55812844 fbshipit-source-id: e0be4c2dc295141d673cccb1aeecee45894b1e70 (cherry picked from commit 599cfde)
Summary: Minor updates to the prerequisite section of the LLM getting started guide. Passing -s to pyenv install prevents a prompt if python 3.10 is already installed (it will just silently continue in this case when the flag is passed). Additionally, under pyenv, we should be using python, not python3. I also added a little bit of wording on env management. Pull Request resolved: #2940 Test Plan: Ran LLM guide prerequisite section on an m1 mac with pyenv-virtualenv. Reviewed By: byjlw Differential Revision: D55913382 Pulled By: GregoryComer fbshipit-source-id: 7f04262b025db83b8621c972c90d3cdc3f029377 (cherry picked from commit 218f643) Co-authored-by: Gregory Comer <gregoryjcomer@gmail.com>
Summary: Version hash reported by https://github.com/facebook/buck2/releases/download/2024-02-15/buck2-x86_64-apple-darwin.zst Pull Request resolved: #2868 Reviewed By: Olivia-liu Differential Revision: D55914146 Pulled By: GregoryComer fbshipit-source-id: b9882900acfd4cb6f74eda90a7c99bdb119ec122 (cherry picked from commit de7fdaa)
) (#2971) Summary: Pull Request resolved: #2952 * Some auto-formatting by my VSCode (remove extra spaces) * Remove imports that have been imported in previous part of the doc * Other minor changes to keep consistency across the doc * Link a screenshot instead of using the raw table because the original table is illegible: {F1482781056} Reviewed By: GregoryComer Differential Revision: D55938344 fbshipit-source-id: 699abb9ebe1196ab73d90a3d08d60be7aa0d8688 (cherry picked from commit e733f2d) Co-authored-by: Olivia Liu <olivialpx@meta.com>
Summary: Pull Request resolved: #2992 We should promote the llama2 page more in https://github.com/pytorch/executorch/tree/main/examples/ bypass-github-export-checks bypass-github-pytorch-ci-checks bypass-github-executorch-ci-checks Reviewed By: iseeyuan Differential Revision: D56018978 fbshipit-source-id: cbbc7bd2ea4ce55e564bd6b4a2900f623599dde6 (cherry picked from commit e641ffc) Co-authored-by: Mergen Nachin <mnachin@meta.com>
These pip dependencies need to be present to build the pip wheel. Also, change the version to a stub that looks less like a real version, until we can hook up the logic to get the version from the git repo state.
Manually install build requirements because `python setup.py bdist_wheel` does not install them.
setup.py is sometimes run as root in docker containers. buck2 doesn't allow running as root unless $HOME is owned by root or does not exist. So temporarily undefine it while configuring cmake, which runs buck2 to get some source lists. Also, the buck2 daemon can sometimes get stuck on the CI workers. Try killing it before starting the build, ignoring any failures.
Some CI jobs can fail with "OS file watch limit reached" when running buck2. This section should reduce the number of files that it tries to watch.
Change the build-wheels workflow to only fetch the first layer of submodules. ExecuTorch only needs the first layer of submodules to build its pip package, but the `build_wheels_*.yaml` workflows will recursively fetch all submodules by default. Fetching all submodules can also cause `buck2` to fail because it will try to watch too many files. This change makes `buck2` work on the CI runners, speeds up the jobs, and reduces disk/network usage.
Always build the pybindings when building the pip wheel. Always link in XNNPACK. On macos, also link in MPS. Core ML can't build on the worker machine, though, because the version of macOS is too old; Core ML requires some features introduced in macOS 10.15.
Passing the `std::` functions directory to unary_ufunc_realhb_to_bool can cause "error: cannot resolve overloaded function ‘isinf’ based on conversion to type ‘torch::executor::FunctionRef<bool(double)>’" in some compilation environments. Might be because these functions can be templatized, or because they became constexpr in C++23.
Summary: This is a no-op Pull Request resolved: #3005 Test Plan: CI Run with `python -m examples.models.llama2.export_llama -c stories110M.pt -p params.json -kv --use_sdpa_with_kv_cache -X` and with `python -m examples.models.llama2.export_llama -c stories110M.pt -p params.json -kv -X` Make sure both work Reviewed By: cccclai Differential Revision: D56048177 Pulled By: mergennachin fbshipit-source-id: 3ac9ac5c34f6fe215de1cfe8b5ddc7aae3635359 (cherry picked from commit 488afc5) Co-authored-by: Mergen Nachin <mnachin@meta.com>
* add more instructions and examples on Delegation (#2973) Summary: Pull Request resolved: #2973 as title. Reviewed By: vmpuri, byjlw Differential Revision: D55988177 fbshipit-source-id: 8cdc953118ecd22e8e9a809f0dd716a30a7fc117 (cherry picked from commit 17c64a3) * replace Executorch with ExecuTorch to fix lint error --------- Co-authored-by: Songhao Jia <gasoonjia@meta.com>
Summary: Pull Request resolved: #3007 Keep llama_transformer.py to look like stock implementation, so that it can be reused everywhere. Do module swap Reviewed By: cccclai Differential Revision: D56048640 fbshipit-source-id: 76de1b09b7f5d79422bb3b32bc830a9a7ecd935c (cherry picked from commit 74eb8b3) Co-authored-by: Mergen Nachin <mnachin@meta.com>
* Add executorch_no_prim_ops target (#2934) Summary: Pull Request resolved: #2934 Currently `libexecutorch.a` always contain prim ops. This becomes a problem when a binary contains 2 "versions" of `libexecutorch.a`, causing a double registration of the prim ops. For example, `libA.so` depends on `libexecutorch.a` and a binary `B` depends on both `libA.so` and `libexecutorch.a`. Since both `libexecutorch.a` and `libA.so` contains prim ops, they will be registered twice. In this PR I created another library `executorch_no_prim_ops` for `libA.so` to depend on. Reviewed By: cccclai, kirklandsign Differential Revision: D55907752 fbshipit-source-id: 755a9b8d5f6f7cf44d011b83bfdc18be6da1aa05 (cherry picked from commit d309e9d) * Fix failing CI jobs caused by #2934 (#2961) Summary: Pull Request resolved: #2961 Fix these 3 CI job failures caused by #2934 (D55907752): * Apple / build-frameworks-ios / macos-job * trunk / test-arm-backend-delegation / linux-job * trunk / test-coreml-delegate / macos-job Reviewed By: kirklandsign Differential Revision: D55950023 fbshipit-source-id: 6166d9112e6d971d042df1400442395d8044c3b3 (cherry picked from commit d993797) * [NOT-CLEAN-CP] Fix 3 CI jobs (#3006) Summary: * [NOT APPLICABLE IN RELEASE] Apple / build-frameworks-ios / macos-job We removed libcustom_ops_lib.a in #2916 so need to remove it from `build_apple_frameworks.sh`. * [NOT APPLICABLE IN RELEASE] Lint / lintrunner / linux-job Remove extra line in backends/qualcomm/quantizer/utils.py * pull / unittest / macos (buck2) / macos-job Fix it by using `executorch_no_prim_ops` instead of `executorch` in MPS and CoreML. Pull Request resolved: #3006 Reviewed By: lucylq Differential Revision: D56048430 Pulled By: larryliu0820 fbshipit-source-id: 9dcb476eea446ea3aba566d595167c691fb00eec (cherry picked from commit 5b7c4ba) --------- Co-authored-by: Mengwei Liu <larryliu@meta.com> Co-authored-by: Mengwei Liu <larryliu@fb.com>
Summary: We have refactors recently and need to update the tutorial and cmake. See #2955 for isseues. Pull Request resolved: #2956 Reviewed By: mcr229, cccclai Differential Revision: D55947725 Pulled By: kirklandsign fbshipit-source-id: f23af28b9a8fe071223d8ffa922a6cd4e49efe61 (cherry picked from commit c7fd394)
Summary: Pull Request resolved: #3254 Create a new page for the new util functions Chen and I made to debug delegations. These functions were well-received within the team as well as by partner teams including modai, thus I think it's important to call them out in our documentation. The examples were copied from the llm manual, but reworded a little bit to flow naturally in this doc. bypass-github-export-checks bypass-github-pytorch-ci-checks bypass-github-executorch-ci-checks Reviewed By: cccclai Differential Revision: D56491214 fbshipit-source-id: 162b4ae75e79730218b0d669d1ec2a7a914b933c (cherry picked from commit bf9888f) Co-authored-by: Olivia Liu <olivialpx@meta.com>
Summary: Pull Request resolved: #3240 The time scale of delegate events reported might be different from the timescale of CPU events. This diff adds support for providing a callable that can be invoked by Inspector to modify the timescale of delegated events to ensure consistency in timescales across delegated and non-delegated events. Reviewed By: Jack-Khuu Differential Revision: D55298701 fbshipit-source-id: e888e51b602c7e1ec8cb9e05ac052280daa12823 (cherry picked from commit b7b40ac) Co-authored-by: Tarun Karuturi <tkaruturi@meta.com>
Summary: Pull Request resolved: #3176 This diff moves llm manual code from outside github (Dave's and Georgey's) to executorch codebase for better pointing to. After this diff. //executorch/examples/llm_maunal will become the only source of truth of our llm manual code. Reviewed By: byjlw, dbort Differential Revision: D56365058 fbshipit-source-id: 97280fc0ca955caabb6056cddbb72102ed711f2c (cherry picked from commit b6e54d0) Co-authored-by: Songhao Jia <gasoonjia@meta.com>
Summary: Pull Request resolved: #3202 This diff adds dynamic export into llm manual, including code and related comments. Also update other documentations for better understanding. Reviewed By: dbort Differential Revision: D56365041 fbshipit-source-id: 5ce4c15206a2923c4d54811cefca03f72869c719 (cherry picked from commit 66a350b) Co-authored-by: Songhao Jia <gasoonjia@meta.com>
* [Android] Fix upload workflow for release * [RELEASE ONLY] Android custom op registration
Fill out the recommended `project` keys, most of which will affect the web page that PyPI will render for the `executorch` package. See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#about-your-project for the latest guidance. Use https://github.com/pytorch/pytorch/blob/a21327e0b03cc18850a0608be2d9c5bd38fd4646/setup.py#L1394 as a guide for the actual values. Add a README-wheel.md file that will be included in the wheel, and will become the main page contents on PyPI. Test Plan: * Installed the package with `./install_requirements.sh` * Looked at the files under ~/miniconda3/envs/executorch/lib/python3.10/site-packages/executorch-0.2.0a0+1a499e0.dist-info. METADATA and LICENSE both contain the new metadata.
Simplify the instruction.
Summary: Use relative markdown links instead of full URLs. This way, the docs will always point to a consistent branch. Pull Request resolved: #3244 Test Plan: Clicked on all modified links in the rendered docs preview: https://docs-preview.pytorch.org/pytorch/executorch/3244/llm/getting-started.html Reviewed By: Gasoonjia Differential Revision: D56479234 Pulled By: dbort fbshipit-source-id: 45fb25f017c73df8606c3fb861acafbdd82fec8c (cherry picked from commit b560864) Co-authored-by: Dave Bort <dbort@meta.com>
* Fix lint Remove `requires-python = ">=3.10"`. This caused the linter to use a new `with` syntax that was added in python 3.10, but we want to eventualy supoort older versions of python. * [ci] Look on pytorch servers when installing pip deps When installing the executorch pip package for CI jobs, look on the pytorch servers when resolving dependencies. This lets the executorch package depend on pytorch pre-release and nightly versions. Also run the llava setup with `-x` to make it easier to debug failures. * [pyproject.toml] Add a dependency on `torch==2.3` This is the version that the release/0.2 version of executorch depends on. We should not pick this back into main.
Summary: Pull Request resolved: #3301 overriding_review_checks_triggers_an_audit_and_retroactive_review Oncall Short Name: executorch Differential Revision: D56517032 fbshipit-source-id: ec2f7fbb1111daf8bd529e0917be698bac3435f4 (cherry picked from commit 5b0030f) Co-authored-by: Anthony Shoumikhin <shoumikhin@meta.com>
Summary: Pull Request resolved: #3300 This diff solves part of Ali's comments in our tracer sheet (https://docs.google.com/spreadsheets/d/1PoJt7P9qMkFSaMmS9f9j8dVcTFhOmNHotQYpwBySydI/edit#gid=0). Specifically speaking： "NanoGPT" -> "nanoGPT" "CoreML" -> "Core ML" "ExecuTorch Codebase" -> "ExecuTorch codebase" "Android Phone" -> "Android phone" "How to build Mobile Apps" -> "How to Build Mobile Apps" also shorten the following two column names for avoid overlapping. "occurrences_in_delegated_graphs" -> "# in_delegated_graphs" "occurrences_in_non_delegated_graphs" -> # in_non_delegated_graphs Reviewed By: Jack-Khuu Differential Revision: D56513601 fbshipit-source-id: 7015c2c5b94b79bc6c57c533ee812c9e58ab9d56 (cherry picked from commit b669056) Co-authored-by: Songhao Jia <gasoonjia@meta.com>
* Stop linking MPS into the prebuilt pip wheel We haven't tested this, and we'd prefer to have both MPS and Core ML working. Remove it for now, putting macOS and Linux on equal footing. * Update README-wheel.md to document what's linked into pybindings Since pybindings can be built with many configurations, it's important to tell users what's actually present in the wheel.
Summary: Will land once we create a new tag
… as Custom Metal kernels are yet not enabled) (#3327)
Summary: - add note for embedding quantize, for llama3 - re-order export args to be the same as llama2, group_size missing `--` Pull Request resolved: #3315 Reviewed By: cccclai Differential Revision: D56528535 Pulled By: lucylq fbshipit-source-id: 4453070339ebdb3d782b45f96fe43d28c7006092 (cherry picked from commit 34f59ed) Co-authored-by: Lucy Qiu <lfq@meta.com>
* Update readme. Summary: . Reviewed By: cccclai Differential Revision: D56532283 fbshipit-source-id: 62d7c9e8583fdb5c9a1b2e781e80799c06682aae (cherry picked from commit ce1e9c1) * Update readme. Summary: . Reviewed By: cccclai Differential Revision: D56535633 fbshipit-source-id: 070a3b0af9dea234f8ae4be01c37c03b4e0a56e6 (cherry picked from commit 035aee4)
Summary: As titled Reviewed By: lucylq Differential Revision: D56532035 (cherry picked from commit 73ad1fb)
Summary: The old screenshot has outdated event block name and event names. New screenshot was taken from a recent real run. bypass-github-export-checks bypass-github-pytorch-ci-checks bypass-github-executorch-ci-checks Reviewed By: tarun292, Jack-Khuu Differential Revision: D56447799 fbshipit-source-id: 040fe45311c9aa8e8a1a0f6756ebda5f0ebbdebf (cherry picked from commit 9c99fe1)
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/3339
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 52aa8cf with merge base d3326a2 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Index_PutSupport, No Need to Skip Anymore (Core ML Has AddedIndex_PutSupport, No Need to Skip Anymore #2975) (Core ML Has AddedIndex_PutSupport, No Need to Skip Anymore #3157)torch==2.3([pyproject.toml] Add a dependency ontorch==2.3#3277)