Code

[gemini] fix ci#5748

Code

Merged

botbw merged 206 commits intohpcaitech:feature/prefetchhpcaitech/ColossalAI:feature/prefetchfrom botbw:prefetchbotbw/ColossalAI:prefetchCopy head branch name to clipboardMay 23, 2024

Commits

Commits on Jan 11, 2024

• [Inference] First PR for rebuild colossal-infer (hpcaitech#5143)

  Show description for 4cf4682
  

  CjhHa1

  authored and

  FrankLeeeee

  committed
  4cf4682

  Copy full SHA for 4cf4682

• [Inference] Add readme (roadmap) and fulfill request handler (hpcaitech#5147)

  Show description for 56e75ee
  

  CjhHa1

  authored and

  FrankLeeeee

  committed
  56e75ee

  Copy full SHA for 56e75ee

• [Inference/NFC] Clean outdated inference tests and deprecated kernels (hpcaitech#5159)

  Show description for 2bb9224
  

  yuanheng-zhao

  authored and

  FrankLeeeee

  committed
  2bb9224

  Copy full SHA for 2bb9224

• [Inference]Add BatchInferState, Sequence and InferConfig (hpcaitech#5149)

  Show description for fab9b93
  

  isky-cd

  authored and

  FrankLeeeee

  committed
  fab9b93

  Copy full SHA for fab9b93

• [Inference] Add CacheBlock and KV-Cache Manager (hpcaitech#5156)

  Show description for 3de2e62
  

  yuanheng-zhao

  authored and

  FrankLeeeee

  committed
  3de2e62

  Copy full SHA for 3de2e62

• [Inference]Update inference config and fix test (hpcaitech#5178)

  Show description for 93aeacc
  

  CjhHa1

  authored and

  FrankLeeeee

  committed
  93aeacc

  Copy full SHA for 93aeacc

• [Inference] Add the logic of the inference engine (hpcaitech#5173)

  Show description for 8daee26
  

  isky-cd

  authored and

  FrankLeeeee

  committed
  8daee26

  Copy full SHA for 8daee26

• [Inference] add logit processor and request handler (hpcaitech#5166)

  Show description for 0e61646
  

  CjhHa1

  authored and

  FrankLeeeee

  committed
  0e61646

  Copy full SHA for 0e61646

• Add padding llama model

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  86853a3

  Copy full SHA for 86853a3

• Fixed a bug in the inference frame

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  62fd08e

  Copy full SHA for 62fd08e

• fix bugs in request_handler

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  6296858

  Copy full SHA for 6296858

• precision alignment

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  9489dc6

  Copy full SHA for 9489dc6

• Fixed a writing error

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  4df8876

  Copy full SHA for 4df8876

• [kernel] Add triton kernel for context attention (FAv2) without padding (hpcaitech#5192)

  Show description for 07b5283
  

  2 peopleauthored and

  FrankLeeeee

  committed
  07b5283

  Copy full SHA for 07b5283

• add context_attention_unpadded

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  02c1bf8

  Copy full SHA for 02c1bf8

• fix bugs in sampler

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  bbfebfb

  Copy full SHA for bbfebfb

• Fixed a typo

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  b2eb9cd

  Copy full SHA for b2eb9cd

• fix beam_width

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  3ad1f3b

  Copy full SHA for 3ad1f3b

• [Inference] Pytorch Attention func, pad&nopad input support (hpcaitech#5219)

  Show description for bfd9b1b
  

  CjhHa1

  authored and

  FrankLeeeee

  committed
  bfd9b1b

  Copy full SHA for bfd9b1b

• fix bugs in attention.py and request_handler.py

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  47e53ea

  Copy full SHA for 47e53ea

• adapted to pad_context_forward

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  fa4fbdb

  Copy full SHA for fa4fbdb

• [Hotfix] Fix accuracy and align attention method api with Triton kernel (hpcaitech#5229)

  Show description for e545a87
  

  CjhHa1

  authored and

  FrankLeeeee

  committed
  e545a87

  Copy full SHA for e545a87

• fix bugs related to processing padding mask

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  2a73e82

  Copy full SHA for 2a73e82

• fix CI bugs

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  fab294c

  Copy full SHA for fab294c

• rm torch.cuda.synchronize

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  10e3c9f

  Copy full SHA for 10e3c9f

• fix bugs in request_handler.py and engine.py

  

  isky-cd

  authored and

  FrankLeeeee

  committed
  d40eb26

  Copy full SHA for d40eb26

• [Inference] Kernel: no pad rotary embedding (hpcaitech#5252)

  Show description for fded91d
  

  CjhHa1

  authored and

  FrankLeeeee

  committed
  fded91d

  Copy full SHA for fded91d

• [kernel] Add flash decoding triton kernel for blocked kv cache (hpcaitech#5249)

  Show description for 1513f20
  

  yuanheng-zhao

  authored and

  FrankLeeeee

  committed
  1513f20

  Copy full SHA for 1513f20

• [git] fixed rebased files

  FrankLeeeee

  committed
  1ded7e8

  Copy full SHA for 1ded7e8

Commits on Jan 15, 2024

• [kernel] Add KV cache copy kernel during decoding (hpcaitech#5261)

  Show description for fa85e02
  yuanheng-zhao

  authored
  fa85e02

  Copy full SHA for fa85e02

• [doc] updated inference readme (hpcaitech#5269)

  FrankLeeeee

  authored
  c597678

  Copy full SHA for c597678

• [Inference] Fix request handler and add recycle logic (hpcaitech#5260)

  Show description for d8db500
  CjhHa1

  authored
  d8db500

  Copy full SHA for d8db500

Commits on Jan 16, 2024

• [kernel] Revise KVCache copy triton kernel API (hpcaitech#5273)

  Show description for 0f2b46a
  yuanheng-zhao

  authored
  0f2b46a

  Copy full SHA for 0f2b46a

Commits on Jan 17, 2024

• [Inference]Adapted to the triton attn kernels (hpcaitech#5264)

  Show description for 86b63f7
  isky-cd

  authored
  86b63f7

  Copy full SHA for 86b63f7

Commits on Jan 18, 2024

• [kernel] Add RMSLayerNorm triton kernel (hpcaitech#5262)

  Show description for 5ae9099
  nkfyz

  authored
  5ae9099

  Copy full SHA for 5ae9099

• [Hotfix] Fix bugs in testing continuous batching (hpcaitech#5270)

  Show description for 9e2342b
  CjhHa1

  authored
  9e2342b

  Copy full SHA for 9e2342b

Commits on Jan 19, 2024

• [kernel/fix] Performance Optimization for Decoding Kernel and Benchmarking (hpcaitech#5274)

  Show description for 6e487e7
  yuanheng-zhao

  authored
  6e487e7

  Copy full SHA for 6e487e7

Commits on Jan 22, 2024

• [inference] Adapted to Rotary Embedding and RMS Norm (hpcaitech#5283)

  Show description for bfff925
  isky-cd

  authored
  bfff925

  Copy full SHA for bfff925

• add utils.py

  isky-cd

  committed
  cea9c86

  Copy full SHA for cea9c86

• Merge pull request hpcaitech#5297 from yuehuayingxueluo/fix_rotary_embedding

  Show description for b785319
  isky-cd

  authored
  b785319

  Copy full SHA for b785319

Commits on Jan 23, 2024

• [Inference] Benchmarking rotary embedding and add a fetch function (hpcaitech#5277)

  Show description for 8e606ec
  CjhHa1

  authored
  8e606ec

  Copy full SHA for 8e606ec

• [Kernel/Fix] Revise flash attention triton kernel API and add benchmark (hpcaitech#5301)

  Show description for 3da9993
  yuanheng-zhao

  authored
  3da9993

  Copy full SHA for 3da9993

Commits on Jan 24, 2024

• [Inference]Add fused rotary kernel and get cos cache kernel (hpcaitech#5302)

  Show description for c647e00
  CjhHa1

  authored
  c647e00

  Copy full SHA for c647e00

Commits on Jan 25, 2024

• [hotfix] fix boundary check in batch (hpcaitech#5306)

  yuanheng-zhao

  authored
  af8359c

  Copy full SHA for af8359c

Commits on Jan 26, 2024

• [inference]Optimize the usage of the mid tensors space in flash attn (hpcaitech#5304)

  Show description for 4f28cb4
  isky-cd

  authored
  4f28cb4

  Copy full SHA for 4f28cb4

• fix (hpcaitech#5311)

  CjhHa1

  authored
  7ddd8b3

  Copy full SHA for 7ddd8b3

Commits on Jan 29, 2024

• [Inference] Update rms norm kernel, benchmark with vLLM (hpcaitech#5315)

  Show description for 1f8a75d
  CjhHa1

  authored
  1f8a75d

  Copy full SHA for 1f8a75d

• [DOC] Update inference readme (hpcaitech#5280)

  Show description for c7c104c
  CjhHa1

  authored
  c7c104c

  Copy full SHA for c7c104c

Commits on Jan 30, 2024

• [Inference]Add Nopadding Llama Modeling (hpcaitech#5327)

  Show description for e8f0642
  isky-cd

  authored
  e8f0642

  Copy full SHA for e8f0642

• [Infer] Optimize Blocked KVCache And Kernels Using It (hpcaitech#5325)

  Show description for 5f98a9d
  yuanheng-zhao

  authored
  5f98a9d

  Copy full SHA for 5f98a9d

Commits on Jan 31, 2024

• merge commit

  FrankLeeeee

  committed
  c565519

  Copy full SHA for c565519

• Merge pull request hpcaitech#5339 from FrankLeeeee/sync/merge-main

  Show description for 1336838
  FrankLeeeee

  authored
  1336838

  Copy full SHA for 1336838

• [Inference] Kernel Fusion, fused copy kv cache into rotary embedding (hpcaitech#5336)

  Show description for df0aa49
  CjhHa1

  authored
  df0aa49

  Copy full SHA for df0aa49

Commits on Feb 1, 2024

• [inference] simplified config verification (hpcaitech#5346)

  Show description for f8e456d
  FrankLeeeee

  authored
  f8e456d

  Copy full SHA for f8e456d

• [Inference]Repalce Attention layer and MLP layer by shardformer to optimize the weight transpose operation，add fused_qkv and fused linear_add (hpcaitech#5340)

  Show description for 249644c
  isky-cd

  authored
  249644c

  Copy full SHA for 249644c

Commits on Feb 2, 2024

• [inference] removed redundancy init_batch (hpcaitech#5353)

  FrankLeeeee

  authored
  db1a763

  Copy full SHA for db1a763

• [inference] moved ops tests to test_infer (hpcaitech#5354)

  FrankLeeeee

  authored
  e76acbb

  Copy full SHA for e76acbb

• [doc] updated inference readme (hpcaitech#5343)

  FrankLeeeee

  authored
  027aa10

  Copy full SHA for 027aa10

• [Inference/opt]Optimize the mid tensor of RMS Norm (hpcaitech#5350)

  Show description for 21ad4a2
  isky-cd

  authored
  21ad4a2

  Copy full SHA for 21ad4a2

• [Inference]Optimize generation process of inference engine (hpcaitech#5356)

  Show description for 631862f
  isky-cd

  authored
  631862f

  Copy full SHA for 631862f

Commits on Feb 6, 2024

• [Fix/Infer] Remove unused deps and revise requirements (hpcaitech#5341)

  Show description for 1dedb57
  yuanheng-zhao

  authored
  1dedb57

  Copy full SHA for 1dedb57

• [Inference]Fused the gate and up proj in mlp，and optimized the autograd process. (hpcaitech#5365)

  Show description for 35382a7
  isky-cd

  authored
  35382a7

  Copy full SHA for 35382a7

Commits on Feb 7, 2024

• [Inference] Adapt to Fused rotary (hpcaitech#5348)

  Show description for 9f4ab2e
  CjhHa1

  authored
  9f4ab2e

  Copy full SHA for 9f4ab2e

• Revert "[Inference] Adapt to Fused rotary (hpcaitech#5348)" (hpcaitech#5373)

  Show description for 8106ede
  FrankLeeeee

  authored
  8106ede

  Copy full SHA for 8106ede

• [inference] added inference template (hpcaitech#5375)

  FrankLeeeee

  authored
  58740b5

  Copy full SHA for 58740b5

• [Inference/opt] Fused KVCahce Memcopy (hpcaitech#5374)

  Show description for 6fb4bcb
  isky-cd

  authored
  6fb4bcb

  Copy full SHA for 6fb4bcb

• [Inference] User Experience: update the logic of default tokenizer and generation config. (hpcaitech#5337)

  Show description for 1f8c7e7
  CjhHa1

  authored
  1f8c7e7

  Copy full SHA for 1f8c7e7

Commits on Feb 8, 2024

• [inference] refactored config (hpcaitech#5376)

  FrankLeeeee

  authored
  9afa520

  Copy full SHA for 9afa520

• [Inference]Support vllm testing in benchmark scripts (hpcaitech#5379)

  Show description for 8c69deb
  isky-cd

  authored
  8c69deb

  Copy full SHA for 8c69deb

Commits on Feb 19, 2024

• [Inference] Optimize and Refactor Inference Batching/Scheduling (hpcaitech#5367)

  Show description for b21aac5
  yuanheng-zhao

  authored
  b21aac5

  Copy full SHA for b21aac5

Commits on Feb 21, 2024

• [Inference]Fused kv copy into rotary calculation (hpcaitech#5383)

  Show description for 7301038
  CjhHa1

  authored
  7301038

  Copy full SHA for 7301038

• Optimized the execution interval time between cuda kernels caused by view and memcopy (hpcaitech#5390)

  Show description for 2a718c8
  isky-cd

  authored
  2a718c8

  Copy full SHA for 2a718c8

Commits on Feb 23, 2024

• [Fix/Inference] Fix format of input prompts and input model in inference engine (hpcaitech#5395)

  Show description for bc1da87
  isky-cd

  authored
  bc1da87

  Copy full SHA for bc1da87

Commits on Feb 26, 2024

• [Infer/Fix] Fix Dependency in test - RMSNorm kernel (hpcaitech#5399)

  Show description for 1906118
  yuanheng-zhao

  authored
  1906118

  Copy full SHA for 1906118

Commits on Feb 28, 2024

• [Inference]Add CUDA KVCache Kernel (hpcaitech#5406)

  Show description for 600881a
  isky-cd

  authored
  600881a

  Copy full SHA for 600881a

• [Inference]Move benchmark-related code to the example directory. (hpcaitech#5408)

  Show description for 0aa27f1
  isky-cd

  authored
  0aa27f1

  Copy full SHA for 0aa27f1

Commits on Mar 4, 2024

• Merge branch 'main' into sync/main

  FrankLeeeee

  committed
  0310b76

  Copy full SHA for 0310b76

• Merge pull request hpcaitech#5424 from FrankLeeeee/sync/main

  Show description for 593a72e
  FrankLeeeee

  authored
  593a72e

  Copy full SHA for 593a72e

Commits on Mar 7, 2024

• add silu_and_mul for infer

  Courtesy-Xs

  committed
  95c2149

  Copy full SHA for 95c2149

Commits on Mar 8, 2024

• [feat] cuda graph support and refactor non-functional api

  LRY89757

  committed
  cefaeb5

  Copy full SHA for cefaeb5

• Merge pull request hpcaitech#5433 from Courtesy-Xs/add_silu_and_mul

  Show description for 2b28b54
  Courtesy-Xs

  authored
  2b28b54

  Copy full SHA for 2b28b54

• add reusable utils for cuda

  Courtesy-Xs

  committed
  a46598a

  Copy full SHA for a46598a

• Merge branch 'feature/colossal-infer' of https://github.com/hpcaitech/ColossalAI into add_gpu_launch_config

  Courtesy-Xs

  committed
  01d289d

  Copy full SHA for 01d289d

• refactor code

  Courtesy-Xs

  committed
  5eb5ff1

  Copy full SHA for 5eb5ff1

• feat rmsnorm cuda kernel and add unittest, benchmark script (hpcaitech#5417)

  SunflowerAries

  authored
  f7aecc0

  Copy full SHA for f7aecc0

Commits on Mar 11, 2024

• [fix] multi graphs capture error

  LRY89757

  committed
  b2c0d9f

  Copy full SHA for b2c0d9f

• [fix] multi graphs capture error

  LRY89757

  committed
  9dec66f

  Copy full SHA for 9dec66f

• [doc] add doc

  LRY89757

  committed
  633e95b

  Copy full SHA for 633e95b

• Merge pull request hpcaitech#5435 from Courtesy-Xs/add_gpu_launch_config

  Show description for 21e1e36
  Courtesy-Xs

  authored
  21e1e36

  Copy full SHA for 21e1e36

• refactor code

  Courtesy-Xs

  committed
  095c070

  Copy full SHA for 095c070

Commits on Mar 12, 2024

• Merge pull request hpcaitech#5445 from Courtesy-Xs/refactor_infer_compilation

  Show description for 368a2aa
  Courtesy-Xs

  authored
  368a2aa

  Copy full SHA for 368a2aa

• optimize rmsnorm: add vectorized elementwise op, feat loop unrolling (hpcaitech#5441)

  SunflowerAries

  authored
  b699f54

  Copy full SHA for b699f54

Commits on Mar 13, 2024

• fix include path

  Courtesy-Xs

  committed
  c1c45e9

  Copy full SHA for c1c45e9

• Merge pull request hpcaitech#5452 from Courtesy-Xs/fix_include_path

  Show description for 6fd355a
  Courtesy-Xs

  authored
  6fd355a

  Copy full SHA for 6fd355a

• fix rmsnorm template function invocation problem(template function partial specialization is not allowed in Cpp) and luckily pass e2e precision test (hpcaitech#5454)

  SunflowerAries

  authored
  ed431de

  Copy full SHA for ed431de

• [Inference/kernel]Add Fused Rotary Embedding and KVCache Memcopy CUDA Kernel (hpcaitech#5418)

  Show description for f366a5e
  isky-cd

  authored
  f366a5e

  Copy full SHA for f366a5e

• [fix] pytest and fix dyn grid bug

  LRY89757

  committed
  1821a6d

  Copy full SHA for 1821a6d

Commits on Mar 14, 2024

• diverse tests

  LRY89757

  committed
  ae24b4f

  Copy full SHA for ae24b4f

• Merge branch 'feature/colossal-infer' into colossal-infer-cuda-graph

  LRY89757

  authored
  d02e257

  Copy full SHA for d02e257

• add implementatino for GetGPULaunchConfig1D

  Courtesy-Xs

  committed
  388e043

  Copy full SHA for 388e043

• [fix] tmp for test

  LRY89757

  committed
  6e30248

  Copy full SHA for 6e30248

Commits on Mar 15, 2024

• add some comments

  Courtesy-Xs

  committed
  5724b9e

  Copy full SHA for 5724b9e

• Merge pull request hpcaitech#5457 from Courtesy-Xs/ly_add_implementation_for_launch_config

  Show description for b6e9785
  Courtesy-Xs

  authored
  b6e9785

  Copy full SHA for b6e9785

Commits on Mar 19, 2024

• refactor vector utils

  Courtesy-Xs

  committed
  48c4f29

  Copy full SHA for 48c4f29

• [feat] add use_cuda_kernel option

  LRY89757

  committed
  aabc9fb

  Copy full SHA for aabc9fb

• Merge pull request hpcaitech#5469 from Courtesy-Xs/add_vec_traits

  Show description for b96557b
  Courtesy-Xs

  authored
  b96557b

  Copy full SHA for b96557b

• add vec_type_trait implementation (hpcaitech#5473)

  Courtesy-Xs

  authored
  7ff42cc

  Copy full SHA for 7ff42cc

Commits on Mar 21, 2024

• [fix] unused option

  LRY89757

  committed
  4eafe0c

  Copy full SHA for 4eafe0c

• Merge branch 'feature/colossal-infer' of https://github.com/hpcaitech/ColossalAI into colossal-infer-cuda-graph

  LRY89757

  committed
  606603b

  Copy full SHA for 606603b

• [fix]

  LRY89757

  committed
  5b017d6

  Copy full SHA for 5b017d6

Commits on Mar 25, 2024

• [fix]

  LRY89757

  committed
  9fe61b4

  Copy full SHA for 9fe61b4

• [fix] remove unused comment

  LRY89757

  committed
  ff4998c

  Copy full SHA for ff4998c

• [Inference]Support FP16/BF16 Flash Attention 2 And Add high_precision Flag To Rotary Embedding (hpcaitech#5461)

  Show description for 87079cf
  isky-cd

  authored
  87079cf

  Copy full SHA for 87079cf

• [fix] merge conflicts

  LRY89757

  committed
  68e9396

  Copy full SHA for 68e9396

• Merge pull request hpcaitech#5434 from LRY89757/colossal-infer-cuda-graph

  Show description for 1d62623
  LRY89757

  authored
  1d62623

  Copy full SHA for 1d62623

• [fix] PR hpcaitech#5354 (hpcaitech#5501)

  Show description for 6251d68
  LRY89757

  authored
  6251d68

  Copy full SHA for 6251d68

Commits on Mar 26, 2024

• [Inference] Optimize request handler of llama (hpcaitech#5512)

  Show description for e6496dd
  Courtesy-Xs

  authored
  e6496dd

  Copy full SHA for e6496dd

Commits on Mar 28, 2024

• The writing style of tail processing and the logic related to macro definitions have been optimized. (hpcaitech#5519)

  isky-cd

  authored
  934e31a

  Copy full SHA for 934e31a

Commits on Apr 1, 2024

• [Inference/Kernel]Add get_cos_and_sin Kernel (hpcaitech#5528)

  Show description for 04aca9e
  isky-cd

  authored
  04aca9e

  Copy full SHA for 04aca9e

• [Inference] Add Reduce Utils (hpcaitech#5537)

  Show description for a2878e3
  Courtesy-Xs

  authored
  a2878e3

  Copy full SHA for a2878e3

Commits on Apr 2, 2024

• [Fix/Inference] Remove unused and non-functional functions (hpcaitech#5543)

  Show description for 4bb5d89
  yuanheng-zhao

  authored
  4bb5d89

  Copy full SHA for 4bb5d89

Commits on Apr 8, 2024

• add cast and op_functor for cuda build-in types (hpcaitech#5546)

  Courtesy-Xs

  authored
  7ebdf48

  Copy full SHA for 7ebdf48

• [Fix] resolve conflicts of merging main

  yuanheng-zhao

  committed
  ed5ebd1

  Copy full SHA for ed5ebd1

• remove unused triton kernels

  yuanheng-zhao

  committed
  ce9401a

  Copy full SHA for ce9401a

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for d788175
  pre-commit-ci[bot]

  committed
  d788175

  Copy full SHA for d788175

• remove outdated triton test

  yuanheng-zhao

  committed
  7ca1d1c

  Copy full SHA for 7ca1d1c

Commits on Apr 9, 2024

• Sync main to feature/colossal-infer

  Show description for d56c963
  yuanheng-zhao

  authored
  d56c963

  Copy full SHA for d56c963

Commits on Apr 10, 2024

• [Infer] Revise and Adapt Triton Kernels for Spec-Dec (hpcaitech#5401)

  Show description for d63c469
  yuanheng-zhao

  committed
  d63c469

  Copy full SHA for d63c469

• [Inference/SpecDec] Add Basic Drafter Model Container (hpcaitech#5405)

  Show description for 5a9b05f
  yuanheng-zhao

  committed
  5a9b05f

  Copy full SHA for 5a9b05f

• [Inference/SpecDec] Add Speculative Decoding Implementation (hpcaitech#5423)

  Show description for a37f826
  yuanheng-zhao

  committed
  a37f826

  Copy full SHA for a37f826

• [SpecDec] Fix inputs for speculation and revise past KV trimming (hpcaitech#5449)

  Show description for 912e24b
  yuanheng-zhao

  committed
  912e24b

  Copy full SHA for 912e24b

• [Inference/SpecDec] Support GLIDE Drafter Model (hpcaitech#5455)

  Show description for d85d914
  yuanheng-zhao

  committed
  d85d914

  Copy full SHA for d85d914

• [doc] Add inference/speculative-decoding README (hpcaitech#5552)

  Show description for e1acb58
  yuanheng-zhao

  committed
  e1acb58

  Copy full SHA for e1acb58

• [Fix] resolve conflicts of rebasing feat/speculative-decoding (hpcaitech#5557)

  Show description for e60d430
  yuanheng-zhao

  committed
  e60d430

  Copy full SHA for e60d430

• [Fix] Llama Modeling Control with Spec-Dec (hpcaitech#5580)

  Show description for f8598e3
  yuanheng-zhao

  committed
  f8598e3

  Copy full SHA for f8598e3

• [Inference/Spec-Dec] Merge pull request hpcaitech#5565 from hpcaitech/feat/speculative-decoding

  Show description for 25928d8
  yuanheng-zhao

  authored
  25928d8

  Copy full SHA for 25928d8

Commits on Apr 11, 2024

• refactor csrc (hpcaitech#5582)

  Courtesy-Xs

  authored
  a219123

  Copy full SHA for a219123

Commits on Apr 15, 2024

• [Inference/Refactor] Delete Duplicated code and refactor vec_copy utils and reduce utils (hpcaitech#5593)

  Show description for d4cb023
  Courtesy-Xs

  authored
  d4cb023

  Copy full SHA for d4cb023

• [inference/model]Adapted to the baichuan2-7B model (hpcaitech#5591)

  Show description for 56b222e
  isky-cd

  authored
  56b222e

  Copy full SHA for 56b222e

Commits on Apr 18, 2024

• [Inference/Kernel] Add Paged Decoding kernel, sequence split within the same thread block (hpcaitech#5531)

  Show description for be396ad
  

  SunflowerAries

  and

  pre-commit-ci[bot]

  authored
  be396ad

  Copy full SHA for be396ad

• [Feat]Tensor Model Parallel Support For Inference (hpcaitech#5563)

  Show description for e37ee2f
  LRY89757

  authored
  e37ee2f

  Copy full SHA for e37ee2f

Commits on Apr 19, 2024

• feat baichuan2 rmsnorm whose hidden size equals to 5120 (hpcaitech#5611)

  SunflowerAries

  authored
  ccf7279

  Copy full SHA for ccf7279

Commits on Apr 23, 2024

• [Fix/Inference] Fix GQA Triton and Support Llama3 (hpcaitech#5624)

  Show description for 5d4c1fe
  

  yuanheng-zhao

  and

  pre-commit-ci[bot]

  authored
  5d4c1fe

  Copy full SHA for 5d4c1fe

• [Fix/Inference]Fix CUDA Rotary Rmbedding GQA (hpcaitech#5623)

  Show description for 12f10d5
  isky-cd

  authored
  12f10d5

  Copy full SHA for 12f10d5

• [example] Update Llama Inference example (hpcaitech#5629)

  Show description for 04863a9
  yuanheng-zhao

  authored
  04863a9

  Copy full SHA for 04863a9

Commits on Apr 24, 2024

• [Inference/Refactor] Refactor compilation mechanism and unified multi hw (hpcaitech#5613)

  Show description for 279300d
  Courtesy-Xs

  authored
  279300d

  Copy full SHA for 279300d

• [Fix/Inference]Fix vllm benchmark (hpcaitech#5630)

  Show description for 90cd522
  isky-cd

  authored
  90cd522

  Copy full SHA for 90cd522

Commits on Apr 25, 2024

• [Inference/Kernel] Optimize paged attention: Refactor key cache layout (hpcaitech#5643)

  Show description for a8fd3b0
  

  SunflowerAries

  and

  pre-commit-ci[bot]

  authored
  a8fd3b0

  Copy full SHA for a8fd3b0

• [Fix] Remove obsolete files - inference (hpcaitech#5650)

  yuanheng-zhao

  authored
  f342a93

  Copy full SHA for f342a93

• [Inference]Adapt to baichuan2 13B (hpcaitech#5614)

  Show description for 3c91e3f
  isky-cd

  authored
  3c91e3f

  Copy full SHA for 3c91e3f

Commits on Apr 26, 2024

• [kernel] Support new KCache Layout - Context Attention Triton Kernel (hpcaitech#5658)

  Show description for 5be590b
  yuanheng-zhao

  authored
  5be590b

  Copy full SHA for 5be590b

• [Inference/Feat] Add kvcache quantization support for FlashDecoding (hpcaitech#5656)

  Courtesy-Xs

  authored
  8ccb671

  Copy full SHA for 8ccb671

Commits on Apr 30, 2024

• [Inference/Feat] Feat quant kvcache step2 (hpcaitech#5674)

  Courtesy-Xs

  authored
  808ee6e

  Copy full SHA for 808ee6e

• [Inference] Adapt Baichuan2-13B TP (hpcaitech#5659)

  Show description for 5f00002
  isky-cd

  authored
  5f00002

  Copy full SHA for 5f00002

• [Inference/Kernel] refactor kvcache manager and rotary_embedding and kvcache_memcpy oper… (hpcaitech#5663)

  Show description for 5cd75ce
  SunflowerAries

  authored
  5cd75ce

  Copy full SHA for 5cd75ce

• [Inference/Feat] Add kvcache quant support for fused_rotary_embedding_cache_copy (hpcaitech#5680)

  Courtesy-Xs

  authored
  ef8e4ff

  Copy full SHA for ef8e4ff

• [inference]Add alibi to flash attn function (hpcaitech#5678)

  Show description for f799631
  isky-cd

  authored
  f799631

  Copy full SHA for f799631

• [Inference] Fix quant bits order (hpcaitech#5681)

  Courtesy-Xs

  authored
  9df016f

  Copy full SHA for 9df016f

Commits on May 3, 2024

• [kernel] Support New KCache Layout - Triton Kernel (hpcaitech#5677)

  Show description for 537a3cb
  

  yuanheng-zhao

  and

  pre-commit-ci[bot]

  authored
  537a3cb

  Copy full SHA for 537a3cb

Commits on May 5, 2024

• [sync] resolve conflicts of merging main

  yuanheng-zhao

  committed
  56ed09a

  Copy full SHA for 56ed09a

• [Fix] Fix & Update Inference Tests (compatibility w/ main)

  yuanheng-zhao

  committed
  8754aba

  Copy full SHA for 8754aba

Commits on May 6, 2024

• [Inference] Remove unnecessary float4_ and rename float8_ to float8 (hpcaitech#5679)

  SunflowerAries

  authored
  725fbd2

  Copy full SHA for 725fbd2

• [Sync] Update from main to feature/colossal-infer (Merge pull request hpcaitech#5685)

  Show description for db7b305
  yuanheng-zhao

  authored
  db7b305

  Copy full SHA for db7b305

• [Inference/Feat] Add quant kvcache support for decode_kv_cache_memcpy (hpcaitech#5686)

  Courtesy-Xs

  authored
  1ace106

  Copy full SHA for 1ace106

Commits on May 7, 2024

• [hotfix] Fix KV Heads Number Assignment in KVCacheManager (hpcaitech#5695)

  Show description for f9afe0a
  yuanheng-zhao

  authored
  f9afe0a

  Copy full SHA for f9afe0a

Commits on May 8, 2024

• [Fix] Fix Inference Example, Tests, and Requirements (hpcaitech#5688)

  Show description for 55cc7f3
  yuanheng-zhao

  authored
  55cc7f3

  Copy full SHA for 55cc7f3

• [hotfix] fix OpenMOE example import path (hpcaitech#5697)

  yuanheng-zhao

  authored
  12e7c28

  Copy full SHA for 12e7c28

• [Inference]Adapt temperature processing logic (hpcaitech#5689)

  Show description for 9c2fe79
  isky-cd

  authored
  9c2fe79

  Copy full SHA for 9c2fe79

• [Inference] Support the logic related to ignoring EOS token (hpcaitech#5693)

  Show description for d482922
  isky-cd

  authored
  d482922

  Copy full SHA for d482922

• [Inference] ADD async and sync Api server using FastAPI (hpcaitech#5396)

  Show description for 69cd7e0
  CjhHa1

  committed
  69cd7e0

  Copy full SHA for 69cd7e0

• [Inference] Finish Online Serving Test, add streaming output api, continuous batching test and example (hpcaitech#5432)

  Show description for de378cd
  CjhHa1

  committed
  de378cd

  Copy full SHA for de378cd

• [Online Server] Chat Api for streaming and not streaming response (hpcaitech#5470)

  Show description for c064032
  CjhHa1

  committed
  c064032

  Copy full SHA for c064032

• [Inference] resolve rebase conflicts

  Show description for 7bbb28e
  CjhHa1

  committed
  7bbb28e

  Copy full SHA for 7bbb28e

• [Inference] Fix bugs and docs for feat/online-server (hpcaitech#5598)

  Show description for 61a1b2e
  CjhHa1

  committed
  61a1b2e

  Copy full SHA for 61a1b2e

• resolve rebase conflicts on Branch feat/online-serving

  CjhHa1

  committed
  bc9063a

  Copy full SHA for bc9063a

Commits on May 9, 2024

• [Inference] Add example test_ci script

  CjhHa1

  committed
  5d9a494

  Copy full SHA for 5d9a494

• Merge pull request hpcaitech#5588 from hpcaitech/feat/online-serving

  Show description for 492520d
  CjhHa1

  authored
  492520d

  Copy full SHA for 492520d

• [Inference/Feat] Add quant kvcache interface (hpcaitech#5700)

  Show description for bfad393
  Courtesy-Xs

  authored
  bfad393

  Copy full SHA for bfad393

Commits on May 10, 2024

• [Inference/Feat] Add convert_fp8 op for fp8 test in the future (hpcaitech#5706)

  Show description for 50104ab
  Courtesy-Xs

  authored
  50104ab

  Copy full SHA for 50104ab

Commits on May 11, 2024

• [Inference]Adapt repetition_penalty and no_repeat_ngram_size (hpcaitech#5708)

  Show description for de4bf3d
  isky-cd

  authored
  de4bf3d

  Copy full SHA for de4bf3d

Commits on May 14, 2024

• [Feat]Inference RPC Server Support (hpcaitech#5705)

  Show description for 18d67d0
  

  LRY89757

  and

  pre-commit-ci[bot]

  authored
  18d67d0

  Copy full SHA for 18d67d0

• add paged-attetionv2: support seq length split across thread block (hpcaitech#5707)

  SunflowerAries

  authored
  7806842

  Copy full SHA for 7806842

• [Inference] Delete duplicated copy_vector (hpcaitech#5716)

  Courtesy-Xs

  authored
  121d7ad

  Copy full SHA for 121d7ad

• [ci] Fix example tests (hpcaitech#5714)

  Show description for 5bbab15
  yuanheng-zhao

  authored
  5bbab15

  Copy full SHA for 5bbab15

• [Fix] Llama3 Load/Omit CheckpointIO Temporarily (hpcaitech#5717)

  Show description for 74c4792
  LRY89757

  authored
  74c4792

  Copy full SHA for 74c4792

Commits on May 15, 2024

• [Inference] Fix API server, test and example (hpcaitech#5712)

  Show description for f47f2fb
  CjhHa1

  authored
  f47f2fb

  Copy full SHA for f47f2fb

Commits on May 16, 2024

• 【Inference] Delete duplicated package (hpcaitech#5723)

  Courtesy-Xs

  authored
  a8d459f

  Copy full SHA for a8d459f

Commits on May 17, 2024

• [example] Update Inference Example (hpcaitech#5725)

  Show description for 8bcfe36
  yuanheng-zhao

  authored
  8bcfe36

  Copy full SHA for 8bcfe36

• [lazy] fix lazy cls init (hpcaitech#5720)

  Show description for 9d83c6d
  flybird11111

  authored
  9d83c6d

  Copy full SHA for 9d83c6d

Commits on May 19, 2024

• [Inference] Fix Inference Generation Config and Sampling (hpcaitech#5710)

  Show description for 283c407
  yuanheng-zhao

  authored
  283c407

  Copy full SHA for 283c407

Commits on May 20, 2024

• [Fix/Inference] Add unsupported auto-policy error message (hpcaitech#5730)

  Show description for bdf9a00
  yuanheng-zhao

  authored
  bdf9a00

  Copy full SHA for bdf9a00

• [doc] Update Inference Readme (hpcaitech#5736)

  Show description for d8b1ea4
  yuanheng-zhao

  authored
  d8b1ea4

  Copy full SHA for d8b1ea4

• [sync] Sync feature/colossal-infer with main

  yuanheng-zhao

  committed
  8633c15

  Copy full SHA for 8633c15

Commits on May 21, 2024

• [Shardformer] Add parallel output for shardformer models(bloom, falcon) (hpcaitech#5702)

  Show description for 22ce873
  Hz188

  authored
  22ce873

  Copy full SHA for 22ce873

• Merge pull request hpcaitech#5737 from yuanheng-zhao/inference/sync/main

  Show description for c06208e
  yuanheng-zhao

  authored
  c06208e

  Copy full SHA for c06208e

• [bug] fix silly bug

  botbw

  committed
  89c3aee

  Copy full SHA for 89c3aee

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 67f1fdf
  

  pre-commit-ci[bot]

  authored and

  botbw

  committed
  67f1fdf

  Copy full SHA for 67f1fdf

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 08efb4c
  pre-commit-ci[bot]

  committed
  08efb4c

  Copy full SHA for 08efb4c

• [chore] add test for prefetch

  botbw

  committed
  3043d15

  Copy full SHA for 3043d15

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for ba21f26
  pre-commit-ci[bot]

  committed
  ba21f26

  Copy full SHA for ba21f26

• [ci] Temporary fix for build on pr (hpcaitech#5741)

  Show description for c2c8c9c
  yuanheng-zhao

  authored
  c2c8c9c

  Copy full SHA for c2c8c9c

• [NFC] Fix code factors on inference triton kernels (hpcaitech#5743)

  yuanheng-zhao

  authored
  bd38fe6

  Copy full SHA for bd38fe6

Commits on May 22, 2024

• [NFC] fix requirements (hpcaitech#5744)

  yuanheng-zhao

  authored
  498f42c

  Copy full SHA for 498f42c

• [Colossal-Inference] (v0.1.0) Merge pull request hpcaitech#5739 from hpcaitech/feature/colossal-infer

  Show description for df67476
  yuanheng-zhao

  authored
  df67476

  Copy full SHA for df67476

Commits on May 23, 2024

• [inference] release (hpcaitech#5747)

  Show description for 4647ec2
  binmakeswell

  authored
  4647ec2

  Copy full SHA for 4647ec2

• Merge remote-tracking branch 'origin/main' into prefetch

  botbw

  committed
  923b4fb

  Copy full SHA for 923b4fb