Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Streamingllm #1489

Merged
merged 35 commits into from
Sep 5, 2024
Merged

Streamingllm #1489

merged 35 commits into from
Sep 5, 2024

Conversation

chenzhuofu
Copy link
Collaborator

@chenzhuofu chenzhuofu commented Sep 5, 2024 •
edited by wmdi
Loading

Description of changes:

  • Support of Streaming LLM
  • Fix some bugs

Related Issues:

Linked Issues:

  • Issue #

Issues closed by this PR:

  • Closes #

This change is Reviewable

@chenzhuofu
feat: avoid patch query
bc4d9f7
@chenzhuofu
chore: separate apply_pos_encoding from compute_qkv
e41f374
@chenzhuofu
chore: remove unused ptr
5783cf1
@chenzhuofu
fix: memory pointer alignment
ea580f7
@chenzhuofu
chore: minor smplification
be93e5c
@chenzhuofu
feat: StreamingCacheInfo
b6bcd4e
@chenzhuofu
feat: add streamingCache-related meta params
f2634a9
@chenzhuofu
chore: more acurate definition
828b1b8
@chenzhuofu
chore: minor
7e71229
@chenzhuofu
feat: add streamingCacheInfo
a2041ea
@chenzhuofu
feat: apply_pos_encoding & update_qkv_cache, add offset control
694cedf
@chenzhuofu
chore: minor rename
810721e
@chenzhuofu
feat: kernel implementation for streaming cache usage
70a4c2d
@chenzhuofu
feat: implement position encoding for streaming cache
8face9c
@chenzhuofu
fix: params should add (de)serialization method
e94598f
@chenzhuofu
chore: reduce kv cache size
f0d56ec
@chenzhuofu
chore: minor
419e0f8
@chenzhuofu
fix: output misalignment
fb31261
@chenzhuofu
chore: minor
fdf7b86
@chenzhuofu
fix: speculative decoding update_custom_mask only consider mask withi…
77aa1af 
…n current batch
@chenzhuofu
fix: barrier_flag initial value
425d770
@chenzhuofu
doc: attention meta info
263d9d6
@chenzhuofu
docs: minor
689dbd6
@chenzhuofu
Added indexing support for streaming cache.
fe5a8ad
@chenzhuofu
Merge branch 'streamingllm' of github.com:flexflow/FlexFlow into stre…
3a1cf30 
…amingllm
@chenzhuofu
docs: minor
fc4c1cd
@chenzhuofu
chore: minor rename
e55cc6e
@chenzhuofu
feat: add streaming-llm logic to attention
8f056af
@chenzhuofu
fix: typo
813e43f
@chenzhuofu
fix: minor bugs in streaming llm
e1477d4
@chenzhuofu chenzhuofu merged commit e42f596 into specscheduler Sep 5, 2024
28 of 32 checks passed
@chenzhuofu chenzhuofu deleted the streamingllm branch September 5, 2024 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zikun-li zikun-li zikun-li approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@chenzhuofu @zikun-li