[WebGPU] Update all wasms to reflect PagedKVCache and ApplyPresenceAndRequencyPenalty #90
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CharlieFRuan
added a commit
to mlc-ai/web-llm
that referenced
this pull request
Feb 15, 2024
This PR adds `GenerationConfig`, which allows per-generation configs.
See `get-started.ts` for its example usage:
```typescript
let genConfig: webllm.GenerationConfig = {
presence_penalty: 0.5,
frequency_penalty: 0.5,
max_gen_len: 20,
// stop: ["is", "Canada"] // for demonstration purpose
}
const prompt0 = "What is the capital of Canada?";
const reply0 = await chat.generate(prompt0, generateProgressCallback, 1, genConfig);
```
In addition to the existing fields in `mlc-chat-config.json`, we also
support OpenAI-like fields `frequency_penalty`, `presence_penalty`, and
`stop` to prepare for the incoming OpenAI-like APIs.
This PR also sets up unit tests; use `npm test` to run tests. However,
some work needs to be done to support end-to-end testing (e.g. accessing
WebGPU in a test environment).
All prebuilt WASMs are updated correspondingly:
mlc-ai/binary-mlc-llm-libs#90 as we introduced a
new API in tvmjs's `runtime.ts` via
apache/tvm#16504.
Note that the update of Llama WASMs is breaking in the sense that users
will have to update their WebLLM npm.
CharlieFRuan
added a commit
to mlc-ai/web-llm
that referenced
this pull request
Feb 15, 2024
The new version includes two main changes: 1. We now support models compiled with `PagedKVCache` (only Llama-variants for now) - This is breaking in the sense that, in order to use the update Llama WASMs, an update to the WebLLM npm is required - WASMs updated here: mlc-ai/binary-mlc-llm-libs#90 - For more see #293 2. We now support `GenerationConfig`, allowing configuring each generation (e.g. repetition penalty, temperature, etc.) - All WASMs needed to be recompiled since we included new function `ApplyPresenceAndRequencyPenalty()` in tvmjs - For more see #298 Other changes include: - #285
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are two main changes that this PR's WASMs include:
ApplyPresenceAndRequencyPenalty()in tvmjs, requiring us to rebuild all wasms.