Code

[Model] Jamba support#4115

Code

Merged

zhuohan123 merged 117 commits intovllm-project:mainvllm-project/vllm:mainfrom mzusman:jamba-supportmzusman/vllm:jamba-supportCopy head branch name to clipboardJul 2, 2024

+1,192-34

Commits

Commits on Apr 16, 2024

• Merged in jurassic-2.5 (pull request vllm-project#1)

  Show description for 337f67a
  

  ErezSC42

  authored and

  Mor Zusman

  committed
  337f67a

  Copy full SHA for 337f67a

• Merged in jamba-3 (pull request vllm-project#4)

  Show description for 0330e14
  

  Mor Zusman

  committed
  0330e14

  Copy full SHA for 0330e14

• Jamba mamba (vllm-project#3)

  Show description for 07cc899
  

  4 peoplecommitted
  07cc899

  Copy full SHA for 07cc899

• Cuda graph (vllm-project#5)

  Show description for 6d336f6
  

  3 peoplecommitted
  6d336f6

  Copy full SHA for 6d336f6

• dtype (vllm-project#6)

  Show description for 00bce1f
  

  mzusman

  and

  Mor Zusman

  committed
  00bce1f

  Copy full SHA for 00bce1f

• N support (vllm-project#8)

  Show description for 39c27b7
  

  3 peoplecommitted
  39c27b7

  Copy full SHA for 39c27b7

• Tensor parallelism (vllm-project#7)

  Show description for 7c75868
  

  3 peoplecommitted
  7c75868

  Copy full SHA for 7c75868

• After merge fixes

  

  Mor Zusman

  committed
  30e6dcd

  Copy full SHA for 30e6dcd

• Clean up

  

  Mor Zusman

  committed
  5c0efdc

  Copy full SHA for 5c0efdc

• Add release mamba cache to executor_base

  

  Mor Zusman

  committed
  19f11f3

  Copy full SHA for 19f11f3

• Add jamba modifications

  

  Mor Zusman

  committed
  1fb817a

  Copy full SHA for 1fb817a

• Add minimun 1 attention layer

  

  Mor Zusman

  committed
  30ae4a1

  Copy full SHA for 30ae4a1

• More fixes

  

  Mor Zusman

  committed
  7bd9c0a

  Copy full SHA for 7bd9c0a

• Delete mamba cache

  

  Mor Zusman

  committed
  d5ac8e8

  Copy full SHA for d5ac8e8

• Jamba padding to the left

  

  Mor Zusman

  committed
  60b49b5

  Copy full SHA for 60b49b5

• Clean up

  

  Mor Zusman

  committed
  c583fe8

  Copy full SHA for c583fe8

• Add import

  

  Mor Zusman

  committed
  c951b7d

  Copy full SHA for c951b7d

• Another clean up

  

  Mor Zusman

  committed
  da6d0f2

  Copy full SHA for da6d0f2

• Align to main

  

  Mor Zusman

  committed
  eb79923

  Copy full SHA for eb79923

• Fix reduce

  

  Mor Zusman

  committed
  919edba

  Copy full SHA for 919edba

• Another fix

  

  Mor Zusman

  committed
  4668566

  Copy full SHA for 4668566

• Black format for jamba

  

  Mor Zusman

  committed
  11a0737

  Copy full SHA for 11a0737

• Formatting

  

  Mor Zusman

  committed
  7e3415e

  Copy full SHA for 7e3415e

• Formatting with format.sh

  

  Mor Zusman

  committed
  adbd2ae

  Copy full SHA for adbd2ae

• Adding to docs and more

  

  Mor Zusman

  committed
  6daf2a2

  Copy full SHA for 6daf2a2

• Add to readme

  

  Mor Zusman

  committed
  7ee927b

  Copy full SHA for 7ee927b

• Adding comments for prefill mamba

  

  Mor Zusman

  committed
  87fa299

  Copy full SHA for 87fa299

• Formating

  

  Mor Zusman

  committed
  8bca3b6

  Copy full SHA for 8bca3b6

Commits on Apr 17, 2024

• Remove mamba-ssm and conv1d from the build system requirements

  

  Mor Zusman

  committed
  b421877

  Copy full SHA for b421877

Commits on Apr 28, 2024

• Remove autoconfig for jamba

  

  Mor Zusman

  committed
  d9c3319

  Copy full SHA for d9c3319

• Move get_attention_num_layers to model_config

  

  Mor Zusman

  committed
  1c0fad8

  Copy full SHA for 1c0fad8

• Merge branch 'main' of https://github.com/vllm-project/vllm into jamba-support-pr

  

  Mor Zusman

  committed
  e831cfc

  Copy full SHA for e831cfc

• Fix in model config

  

  Mor Zusman

  committed
  1033904

  Copy full SHA for 1033904

• Formatting

  

  Mor Zusman

  committed
  2b93182

  Copy full SHA for 2b93182

Commits on May 5, 2024

• Add layers_block_type_support to model config

  

  Mor Zusman

  committed
  b2f86f8

  Copy full SHA for b2f86f8

• Update Jamba to support changes from main

  

  Mor Zusman

  committed
  7061df7

  Copy full SHA for 7061df7

• Take Jamba config off since its now in transformers

  

  Mor Zusman

  committed
  054faf1

  Copy full SHA for 054faf1

• Take jamba config off

  

  Mor Zusman

  committed
  fb3fc83

  Copy full SHA for fb3fc83

• Format

  

  Mor Zusman

  committed
  6d8765d

  Copy full SHA for 6d8765d

• Refactor the model runner a little , make it more readable and chage

  Show description for 10896ae
  

  Mor Zusman

  committed
  10896ae

  Copy full SHA for 10896ae

• rename release mamba to release seqlen agnostic

  

  Mor Zusman

  committed
  d1dc26f

  Copy full SHA for d1dc26f

• Move requirements of mamba to its own requirements

  

  Mor Zusman

  committed
  07c8cd2

  Copy full SHA for 07c8cd2

• Remove mamba metadata since its mamba specific

  

  Mor Zusman

  committed
  5c11285

  Copy full SHA for 5c11285

• Align with master

  

  Mor Zusman

  committed
  2bb3360

  Copy full SHA for 2bb3360

• Change comment

  

  Mor Zusman

  committed
  a235c44

  Copy full SHA for a235c44

Commits on May 6, 2024

• (1) implement contains_seqlen_agnostic_layers with use of self.get_num_seqlen_agnostic_layers (2) renaming + remove commented code

  tomeras91

  committed
  af7a4ac

  Copy full SHA for af7a4ac

• Jamba official hf (vllm-project#14)

  Show description for 988718e
  tomeras91

  committed
  988718e

  Copy full SHA for 988718e

Commits on May 7, 2024

• fixes missed in merge with official Jamba HF format

  tomeras91

  committed
  49ce3df

  Copy full SHA for 49ce3df

Commits on May 16, 2024

• Merge with main

  zhuohan123

  committed
  4fa065f

  Copy full SHA for 4fa065f

Commits on May 17, 2024

• fix merge error

  zhuohan123

  committed
  7add09a

  Copy full SHA for 7add09a

Commits on May 20, 2024

• Fix bug where seqlen agnostic cache wasn't copied for non driver workers

  

  Mor Zusman

  committed
  14fbab5

  Copy full SHA for 14fbab5

Commits on Jun 25, 2024

• WIP - encapsulate Jamba cache managemnt inside the modeling file

  

  Mor Zusman

  committed
  e3dec15

  Copy full SHA for e3dec15

• Cleanup

  

  Mor Zusman

  committed
  92778c4

  Copy full SHA for 92778c4

Commits on Jun 26, 2024

• Typos and cleanup

  

  Mor Zusman

  committed
  db36427

  Copy full SHA for db36427

• Another typo

  

  Mor Zusman

  committed
  7f6edfc

  Copy full SHA for 7f6edfc

• Keep the finished requests ids after in the scheduler after each step and send them to the workers on every step

  

  Mor Zusman

  committed
  ee5f058

  Copy full SHA for ee5f058

• Cleanup

  

  Mor Zusman

  committed
  2d42367

  Copy full SHA for 2d42367

• clean up requests after profile

  

  Mor Zusman

  committed
  6a6378c

  Copy full SHA for 6a6378c

• Update mamba requirements

  

  Mor Zusman

  committed
  1a8e2f9

  Copy full SHA for 1a8e2f9

• Renaming

  

  Mor Zusman

  committed
  eb89987

  Copy full SHA for eb89987

• Renaming

  

  Mor Zusman

  committed
  1cb8c1c

  Copy full SHA for 1cb8c1c

• Rename and docs

  

  Mor Zusman

  committed
  feca5d5

  Copy full SHA for feca5d5

• Format

  

  Mor Zusman

  committed
  72c31cc

  Copy full SHA for 72c31cc

• Add mamba to Dockerfile

  

  Mor Zusman

  committed
  ddeb689

  Copy full SHA for ddeb689

• Mamba disable prompt batching

  

  Mor Zusman

  committed
  5d5a3be

  Copy full SHA for 5d5a3be

• Format

  

  Mor Zusman

  committed
  85715fe

  Copy full SHA for 85715fe

• Merge branch 'gh-main' into jamba-support-pr

  

  Mor Zusman

  committed
  84aa88f

  Copy full SHA for 84aa88f

• Fix jamba bug

  

  Mor Zusman

  committed
  8c6d82d

  Copy full SHA for 8c6d82d

• Renaming

  

  Mor Zusman

  committed
  628eec7

  Copy full SHA for 628eec7

• WIP - Merge with main adaptations

  

  Mor Zusman

  committed
  30030ce

  Copy full SHA for 30030ce

• Fix

  

  Mor Zusman

  committed
  3fba9bc

  Copy full SHA for 3fba9bc

• Formating

  

  Mor Zusman

  committed
  45f3d96

  Copy full SHA for 45f3d96

Commits on Jun 27, 2024

• deploy the finihsed request ids inside the modelinputs instead of worker

  Show description for 794f1c3
  

  Mor Zusman

  committed
  794f1c3

  Copy full SHA for 794f1c3

• fix

  

  Mor Zusman

  committed
  33eb405

  Copy full SHA for 33eb405

• Renaming

  

  Mor Zusman

  committed
  25c03e7

  Copy full SHA for 25c03e7

• Format

  

  Mor Zusman

  committed
  94d40a8

  Copy full SHA for 94d40a8

• Typing and format

  

  Mor Zusman

  committed
  976166f

  Copy full SHA for 976166f

• Cleanup

  

  Mor Zusman

  committed
  8181821

  Copy full SHA for 8181821

• Remove requirements-common and cuda from requirements-mamba

  

  Mor Zusman

  committed
  4fdc35b

  Copy full SHA for 4fdc35b

• Fix

  

  Mor Zusman

  committed
  aadeca2

  Copy full SHA for aadeca2

• set finished requests ids as none on default

  

  Mor Zusman

  committed
  fee775e

  Copy full SHA for fee775e

• get attr to get num hidden layers

  

  Mor Zusman

  committed
  668f3d9

  Copy full SHA for 668f3d9

• Add jamba test

  

  Mor Zusman

  committed
  10a44dc

  Copy full SHA for 10a44dc

• Ignore jamba test in cpu

  

  Mor Zusman

  committed
  cd9ba35

  Copy full SHA for cd9ba35

• Cleanup

  

  Mor Zusman

  committed
  6df4f69

  Copy full SHA for 6df4f69

• Format and rename

  

  Mor Zusman

  committed
  75dd84e

  Copy full SHA for 75dd84e

• Format

  

  Mor Zusman

  committed
  577f678

  Copy full SHA for 577f678

Commits on Jun 29, 2024

• change num_layers to num_attention_layers and add comment

  

  Mor Zusman

  committed
  7bb332e

  Copy full SHA for 7bb332e

• Extended the finished reqeusts ids comment

  

  Mor Zusman

  committed
  c051758

  Copy full SHA for c051758

• Format and make the jamba code more readable, adding comments and

  Show description for b6dc237
  

  Mor Zusman

  committed
  b6dc237

  Copy full SHA for b6dc237

• Merge branch 'gh-main' into jamba-support-pr

  

  Mor Zusman

  committed
  24b4bf2

  Copy full SHA for 24b4bf2

Commits on Jun 30, 2024

• Format

  

  Mor Zusman

  committed
  b0b0836

  Copy full SHA for b0b0836

• Resolve conflicts and format

  

  Mor Zusman

  committed
  e52e4d7

  Copy full SHA for e52e4d7

• Add finished requests ids to the prepare model spec decoding

  

  Mor Zusman

  committed
  b4d49e0

  Copy full SHA for b4d49e0

• Format

  

  Mor Zusman

  committed
  68e27de

  Copy full SHA for 68e27de

• Test cleanup

  

  Mor Zusman

  committed
  670ff3a

  Copy full SHA for 670ff3a

• Add message to test

  

  Mor Zusman

  committed
  b7e31e3

  Copy full SHA for b7e31e3

Commits on Jul 1, 2024

• Add docstring in vllm/config.py

  

  Mor Zusman

  committed
  571f63d

  Copy full SHA for 571f63d

• rename flush to get_and_reset

  

  Mor Zusman

  committed
  49da326

  Copy full SHA for 49da326

• Add comments

  

  Mor Zusman

  committed
  688732e

  Copy full SHA for 688732e

• Change to private and check finished through all of the queue

  

  Mor Zusman

  committed
  4a6b170

  Copy full SHA for 4a6b170

• CI

  

  Mor Zusman

  committed
  2047a91

  Copy full SHA for 2047a91

• Merge branch 'gh-main' into jamba-support-pr

  

  Mor Zusman

  committed
  f2c407f

  Copy full SHA for f2c407f

• Pipeline Parallelism

  Show description for 5d932a4
  andoorve

  committed
  5d932a4

  Copy full SHA for 5d932a4

• Make scheduler use pipeline parallel size directly

  Show description for 3c15001
  andoorve

  committed
  3c15001

  Copy full SHA for 3c15001

• Change ABC defaults for prepare_model_input

  Show description for 1ff2cdb
  andoorve

  committed
  1ff2cdb

  Copy full SHA for 1ff2cdb

• Add basic comm ops tests with TP and PP.

  Show description for 548f4e8
  andoorve

  committed
  548f4e8

  Copy full SHA for 548f4e8

• Fix phi3v for test

  Show description for 5a4b323
  andoorve

  committed
  5a4b323

  Copy full SHA for 5a4b323

Commits on Jul 2, 2024

• Address Nick nits and fix CUDAGraph correctness

  Show description for c92257c
  andoorve

  committed
  c92257c

  Copy full SHA for c92257c

• Merge branch 'pipeline-parallel' into jamba-support-pr

  

  Mor Zusman

  committed
  60bb1a7

  Copy full SHA for 60bb1a7

• Merge branch 'gh-main' into jamba-support-pr

  

  Mor Zusman

  committed
  2ea2b80

  Copy full SHA for 2ea2b80

• Formating and fixing llm engine

  

  Mor Zusman

  committed
  10d8f3c

  Copy full SHA for 10d8f3c

• Align with main and format

  

  Mor Zusman

  committed
  1331a8f

  Copy full SHA for 1331a8f

• Fix bug

  

  Mor Zusman

  committed
  21c92b4

  Copy full SHA for 21c92b4

• Format

  

  Mor Zusman

  committed
  726ccad

  Copy full SHA for 726ccad

• Add intermediate tensors

  

  Mor Zusman

  committed
  4b6a491

  Copy full SHA for 4b6a491

• Format

  

  Mor Zusman

  committed
  da5d94a

  Copy full SHA for da5d94a