Code

ggml : add Flash Attention#5021

Code

Merged

ggerganov merged 145 commits intomasterggerganov/llama.cpp:masterfrom gg/flash-attnggerganov/llama.cpp:gg/flash-attnCopy head branch name to clipboardApr 30, 2024

+2,917-453

Commits

Commits on Jan 18, 2024

• ggml : add ggml_flash_attn_ext API

  ggerganov

  committed
  a1c004e

  Copy full SHA for a1c004e

Commits on Jan 19, 2024

• ggml : fix GQA support in ggml_flash_attn_ext

  ggerganov

  committed
  fa7ebcc

  Copy full SHA for fa7ebcc

Commits on Jan 20, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  c3cdfff

  Copy full SHA for c3cdfff

• ggml : online attention (CPU)

  ggerganov

  committed
  a9681fe

  Copy full SHA for a9681fe

Commits on Jan 21, 2024

• metal : initial implementation

  ggerganov

  committed
  1173f49

  Copy full SHA for 1173f49

• metal : f16 precision

  ggerganov

  committed
  528da75

  Copy full SHA for 528da75

• metal : reduce branches

  ggerganov

  committed
  52ae085

  Copy full SHA for 52ae085

• metal : specialize for head size

  ggerganov

  committed
  b973258

  Copy full SHA for b973258

• wip : 8 rows per simd group

  ggerganov

  committed
  8cde449

  Copy full SHA for 8cde449

• wip : 4 rows per simd group

  ggerganov

  committed
  f31955f

  Copy full SHA for f31955f

• wip : template for rows per warp

  ggerganov

  committed
  a4b6341

  Copy full SHA for a4b6341

• metal : parallelize across KV size

  ggerganov

  committed
  77d08f3

  Copy full SHA for 77d08f3

• metal : parallel reduce across heads

  ggerganov

  committed
  17720fa

  Copy full SHA for 17720fa

Commits on Jan 25, 2024

• metal : efficient flash_attn_f16 implementation

  ggerganov

  committed
  1446a12

  Copy full SHA for 1446a12

• metal : avoid redundant loads of the attention

  ggerganov

  committed
  d917746

  Copy full SHA for d917746

• metal : scale and mask in matrix form

  ggerganov

  committed
  432ad04

  Copy full SHA for 432ad04

• metal : fix comment

  ggerganov

  committed
  40ea8cd

  Copy full SHA for 40ea8cd

• llama : avoid ggml_cast, use F32 query

  ggerganov

  committed
  f9ca5dc

  Copy full SHA for f9ca5dc

• metal : add parallel reduce version (disabled)

  ggerganov

  committed
  6fea843

  Copy full SHA for 6fea843

Commits on Jan 28, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  b3dd7d9

  Copy full SHA for b3dd7d9

• metal : move output into local memory + optimize

  Show description for 77f6976
  ggerganov

  committed
  77f6976

  Copy full SHA for 77f6976

• metal : add tests, fix scaling, support C > 32

  ggerganov

  committed
  ecc466a

  Copy full SHA for ecc466a

• metal : improve precision

  ggerganov

  committed
  3a428a1

  Copy full SHA for 3a428a1

• ggml : fix f16 mad

  ggerganov

  committed
  8612864

  Copy full SHA for 8612864

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  0ad44ba

  Copy full SHA for 0ad44ba

• metal : minor

  ggerganov

  committed
  134c81c

  Copy full SHA for 134c81c

• metal : support Q > 8

  ggerganov

  committed
  1db22d7

  Copy full SHA for 1db22d7

Commits on Jan 29, 2024

• tests : add ATTN tests

  ggerganov

  committed
  4794821

  Copy full SHA for 4794821

• metal : disable buffer allocation logs

  ggerganov

  committed
  abeaf0d

  Copy full SHA for abeaf0d

• tests : more

  ggerganov

  committed
  c6c1132

  Copy full SHA for c6c1132

• metal : faster inner loop for C == 32

  ggerganov

  committed
  5fcb9c1

  Copy full SHA for 5fcb9c1

Commits on Jan 30, 2024

• metal : fix array initialization

  ggerganov

  committed
  d073e4f

  Copy full SHA for d073e4f

• tests : ifdef

  ggerganov

  committed
  78df552

  Copy full SHA for 78df552

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  3d03bcb

  Copy full SHA for 3d03bcb

Commits on Jan 31, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  2ddc9bb

  Copy full SHA for 2ddc9bb

• ggml : switch to padded F16 mask for ggml_soft_max, ggml_flash_attn_ext

  ggerganov

  committed
  8ad92dc

  Copy full SHA for 8ad92dc

Commits on Feb 1, 2024

• ggml : fix ggml_soft_max mask requirement

  ggerganov

  committed
  910b15b

  Copy full SHA for 910b15b

• cuda : fix soft_max to use correct mask size

  ggerganov

  committed
  2e46013

  Copy full SHA for 2e46013

• cuda : add flash_attn kernel (wip)

  ggerganov

  committed
  5a19a9f

  Copy full SHA for 5a19a9f

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  41d136b

  Copy full SHA for 41d136b

• metal : optimize softmax for C > 32

  ggerganov

  committed
  56e45a2

  Copy full SHA for 56e45a2

• metal : optimize softmax

  ggerganov

  committed
  cda5a60

  Copy full SHA for cda5a60

• tests : minor fix

  ggerganov

  committed
  c6769b9

  Copy full SHA for c6769b9

• cuda : avoid zeroing fragments

  ggerganov

  committed
  db1f3c4

  Copy full SHA for db1f3c4

Commits on Feb 2, 2024

• tests : update dims

  ggerganov

  committed
  12eaa22

  Copy full SHA for 12eaa22

• cuda : fix __hisinf() result check

  ggerganov

  committed
  b68a112

  Copy full SHA for b68a112

Commits on Feb 3, 2024

• cuda : avoid warp_reduce for smax

  ggerganov

  committed
  b150abe

  Copy full SHA for b150abe

• cuda : use int instead of int64_t

  Show description for 7c34655
  ggerganov

  committed
  7c34655

  Copy full SHA for 7c34655

• cuda : make loops use the same loop values

  Show description for 1f8a592
  ggerganov

  committed
  1f8a592

  Copy full SHA for 1f8a592

• cuda : unroll some of the loops

  ggerganov

  committed
  92472ea

  Copy full SHA for 92472ea

• cuda : avoid __hisinf branches

  ggerganov

  committed
  c51f27c

  Copy full SHA for c51f27c

• cuda : use half2 in softmax

  ggerganov

  committed
  b958151

  Copy full SHA for b958151

• cuda : switch to 1 warp for bs > 16

  ggerganov

  committed
  a7b4715

  Copy full SHA for a7b4715

• cuda : speed-up reduce part of the kernel

  ggerganov

  committed
  3b1c4e7

  Copy full SHA for 3b1c4e7

• cuda : unroll Q*K^T loop

  ggerganov

  committed
  5b263dd

  Copy full SHA for 5b263dd

• cuda : fix -INF block check

  ggerganov

  committed
  e04ff39

  Copy full SHA for e04ff39

• cuda : simplify softmax

  ggerganov

  committed
  cfd9732

  Copy full SHA for cfd9732

• cuda : fix matrix names

  ggerganov

  committed
  ef68fac

  Copy full SHA for ef68fac

Commits on Feb 4, 2024

• cuda : minor

  ggerganov

  committed
  1846e92

  Copy full SHA for 1846e92

Commits on Feb 12, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  6875997

  Copy full SHA for 6875997

Commits on Feb 19, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  31109ca

  Copy full SHA for 31109ca

• llama : adapt to F16 KQ_pos

  ggerganov

  committed
  f249c99

  Copy full SHA for f249c99

Commits on Mar 3, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  02a645e

  Copy full SHA for 02a645e

• llama : adapt new models to F16 KQ_mask

  ggerganov

  committed
  6aefd11

  Copy full SHA for 6aefd11

Commits on Mar 4, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  e307882

  Copy full SHA for e307882

• ggml : fix F16 store (ARM NEON)

  ggerganov

  committed
  58c7f61

  Copy full SHA for 58c7f61

Commits on Mar 22, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  9495d39

  Copy full SHA for 9495d39

• llama : fix type of KQ_mask and KQ_pos

  ggerganov

  committed
  3a468e6

  Copy full SHA for 3a468e6

• ggml : fix CPU soft_max

  ggerganov

  committed
  0953212

  Copy full SHA for 0953212

Commits on Mar 24, 2024

• tests : add hs=256

  ggerganov

  committed
  e425810

  Copy full SHA for e425810

Commits on Mar 27, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  013721d

  Copy full SHA for 013721d

• cuda : fix build

  ggerganov

  committed
  6be02b5

  Copy full SHA for 6be02b5

Commits on Mar 28, 2024

• metal : improve perf via smaller int registers

  ggerganov

  committed
  57c03b7

  Copy full SHA for 57c03b7

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  3e318e7

  Copy full SHA for 3e318e7

• cuda : adapt soft_max to F16 mask and pos

  ggerganov

  committed
  08e69c5

  Copy full SHA for 08e69c5

Commits on Apr 2, 2024

• CUDA: faster FlashAttention, kernel for bs == 1

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  75aa7b4

  Copy full SHA for 75aa7b4

• 16 cols for Phi-2

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  d59ac67

  Copy full SHA for d59ac67

• no vec for hs, no hs==256 ncols==32 for Volta

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  81da919

  Copy full SHA for 81da919

• adjust kernel selection logic

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  269374e

  Copy full SHA for 269374e

• 4 warps, 256 stride for all D

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  cca6d02

  Copy full SHA for cca6d02

• no ncols == 64

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  68d793b

  Copy full SHA for 68d793b

• Multiple parallel blocks for batch size 1

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  3f777ac

  Copy full SHA for 3f777ac

• fix compile warnings

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  e1ecd3b

  Copy full SHA for e1ecd3b

• fix excessive KQ_b loads

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  bb0d51a

  Copy full SHA for bb0d51a

• fix cmake build

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  c63dfdf

  Copy full SHA for c63dfdf

• fix KV cache padding, NaN from INFINITY (#6438)

  JohannesGaessler

  authored
  ee19a4a

  Copy full SHA for ee19a4a

Commits on Apr 5, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  89961de

  Copy full SHA for 89961de

Commits on Apr 17, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  2c41180

  Copy full SHA for 2c41180

• llama : flash_attn cparam + fix defrag

  ggerganov

  committed
  599ce84

  Copy full SHA for 599ce84

• server: support flash_attn param

  phymbert

  committed
  4053857

  Copy full SHA for 4053857

• server: bench: enable flash_attn param

  phymbert

  committed
  5668c79

  Copy full SHA for 5668c79

Commits on Apr 18, 2024

• CUDA: refactor host code, dyn. par. blocks

  JohannesGaessler

  committed
  34f93bb

  Copy full SHA for 34f93bb

• fix flash_attn_vec_f16 race condition

  JohannesGaessler

  committed
  6a3b842

  Copy full SHA for 6a3b842

• flush softmax exp below threshold to 0

  JohannesGaessler

  committed
  ef9e159

  Copy full SHA for ef9e159

• store temp KQ in registers

  JohannesGaessler

  committed
  a5b0e2d

  Copy full SHA for a5b0e2d

• Calculate KQ as FP32 if KQV has GGML_PREC_F32

  JohannesGaessler

  committed
  0bc67dd

  Copy full SHA for 0bc67dd

• Add __hgt2_mask implementation for CUDA 11

  JohannesGaessler

  committed
  2f538b9

  Copy full SHA for 2f538b9

• fix KQ FP32 precision fpr parallel_blocks > 1

  JohannesGaessler

  committed
  87968de

  Copy full SHA for 87968de

• llama-bench : add -fa,--flash-attn arg

  ggerganov

  committed
  260cdb2

  Copy full SHA for 260cdb2

• metal : add BS=1 kernel for flash attention (#6508)

  Show description for 105332c
  ggerganov

  authored
  105332c

  Copy full SHA for 105332c

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  fa9e8c6

  Copy full SHA for fa9e8c6

• metal : use F32 attention accumulators

  ggerganov

  committed
  c16a7c2

  Copy full SHA for c16a7c2

• batched-bench : add fattn arg

  ggerganov

  committed
  9ca8698

  Copy full SHA for 9ca8698

Commits on Apr 19, 2024

• llama : simplify llama_build_kv_store

  Show description for 74d57f9
  ggerganov

  committed
  74d57f9

  Copy full SHA for 74d57f9

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  1db66c1

  Copy full SHA for 1db66c1

• llama : adapt build_olmo to changes

  ggerganov

  committed
  e32b281

  Copy full SHA for e32b281

• ggml : fix arm fp16 store on windows

  ggerganov

  committed
  703c6e6

  Copy full SHA for 703c6e6

• metal : clean-up

  ggerganov

  committed
  97eaece

  Copy full SHA for 97eaece

• metal : clean-up kernel code

  ggerganov

  committed
  1a88565

  Copy full SHA for 1a88565

• metal : minor

  ggerganov

  committed
  bc34616

  Copy full SHA for bc34616

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  29f6ad8

  Copy full SHA for 29f6ad8

• tests : remove benchmarks

  Show description for 5294542
  ggerganov

  committed
  5294542

  Copy full SHA for 5294542

• ggml : fix avx512 const correctness

  Show description for 3badef1
  ggerganov

  committed
  3badef1

  Copy full SHA for 3badef1

• ggml : fix soft_max with bias on CPU

  Show description for 871fcb6
  ggerganov

  committed
  871fcb6

  Copy full SHA for 871fcb6

Commits on Apr 22, 2024

• common : print --flash-attn in help

  ggerganov

  committed
  a39217d

  Copy full SHA for a39217d

• ggml : fix num dimensions in ggml_flash_attn_ext

  ggerganov

  committed
  cb76d74

  Copy full SHA for cb76d74

• llama : force disable flash attention for incompatible models

  ggerganov

  committed
  c11d05f

  Copy full SHA for c11d05f

• ggml : ggml_soft_max support F16/F32 mask/pos

  Show description for f725ca9
  ggerganov

  committed
  f725ca9

  Copy full SHA for f725ca9

• cuda : uint -> uint32_t

  ggerganov

  committed
  5408d55

  Copy full SHA for 5408d55

• cuda : "constexpr dim3" -> "const dim3"

  Show description for c70bfd7
  ggerganov

  committed
  c70bfd7

  Copy full SHA for c70bfd7

Commits on Apr 23, 2024

• cuda : try to fix __hgt2_mask

  Show description for c129369
  ggerganov

  committed
  c129369

  Copy full SHA for c129369

• ggml : add TODO's for F16/F32 mask/pos support in other backends

  ggerganov

  committed
  3864eea

  Copy full SHA for 3864eea

• llama : replace bool need_kq_pos with use_alibi

  ggerganov

  committed
  78d363b

  Copy full SHA for 78d363b

• llama : prep ALiBi support for BERT models

  Show description for 19e8982
  ggerganov

  committed
  19e8982

  Copy full SHA for 19e8982

• llama : fix n_batch requirements

  Show description for 56657e5
  ggerganov

  committed
  56657e5

  Copy full SHA for 56657e5

• cont

  ggerganov

  committed
  d228bf8

  Copy full SHA for d228bf8

• server : add help for --flash-attn arg

  ggerganov

  committed
  751591d

  Copy full SHA for 751591d

Commits on Apr 24, 2024

• Merge branch 'master' into gg/flash-attn

  Show description for 8937ec5
  ggerganov

  committed
  8937ec5

  Copy full SHA for 8937ec5

• llama : disable FA for AMD

  ggerganov

  committed
  ce281b9

  Copy full SHA for ce281b9

Commits on Apr 25, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  1f77f49

  Copy full SHA for 1f77f49

• tests : remove TMP_ATTN_BENCH

  Show description for ff2c64a
  ggerganov

  committed
  ff2c64a

  Copy full SHA for ff2c64a

• Merge branch 'master' into gg/flash-attn

  Show description for cb3547a
  ggerganov

  committed
  cb3547a

  Copy full SHA for cb3547a

• llama : support save/load state with FA enabled

  Show description for 1fd5bc3
  ggerganov

  committed
  1fd5bc3

  Copy full SHA for 1fd5bc3

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  09d0381

  Copy full SHA for 09d0381

• ci : add CUDA save-load-state tests

  Show description for ac1c6d9
  ggerganov

  committed
  ac1c6d9

  Copy full SHA for ac1c6d9

• llama : llama_kv_cache_clear zeroes data + fix save-load seq

  Show description for c225609
  ggerganov

  committed
  c225609

  Copy full SHA for c225609

• llama : fix copy-paste errors, add TODO

  ggerganov

  committed
  bab346b

  Copy full SHA for bab346b

• llama : disallow incompatible states

  ggerganov

  committed
  0fc5c5e

  Copy full SHA for 0fc5c5e

• llama : update llama_state_get_size after v_trans field

  ggerganov

  committed
  1e590ac

  Copy full SHA for 1e590ac

• metal : remove tmp log

  ggerganov

  committed
  4f4c024

  Copy full SHA for 4f4c024

• llama : add static reminder for llama_state_get_size

  ggerganov

  committed
  9e38760

  Copy full SHA for 9e38760

Commits on Apr 29, 2024

• Merge branch 'master' into gg/flash-attn

  Show description for a1616e9
  ggerganov

  committed
  a1616e9

  Copy full SHA for a1616e9

• Merge branch 'master' into gg/flash-attn

  Show description for ca0275c
  ggerganov

  committed
  ca0275c

  Copy full SHA for ca0275c

Commits on Apr 30, 2024

• metal : fix max nsg

  Show description for e180fcd
  ggerganov

  committed
  e180fcd

  Copy full SHA for e180fcd

• ci : fix arg order

  Show description for c240ae2
  ggerganov

  committed
  c240ae2

  Copy full SHA for c240ae2