[Feature] Blazing fast W4A16 inference

[lzhangzz]

Preliminary benchmark on Llama-2-7B and A100 GPU (context length=1, generate=512)

• throughput (token/s)

batch size	w4	w16
1	238.61	97.61
4	903.89	379.22
8	1662.58	735.44
16	2799.18	1374.98
32	3779.73	2408.16
64	4761.80	3878.41
128	5638.81	4918.97

512 (instead of 2048) tokens are generated to focus on GEMM workload (instead of Attention which consists of GEMVs mostly).

[lvhan028]

May fix linting

[lvhan028]

May add news in README

[lvhan028]

@irexyc Please help check if it works on windows-3080

[lvhan028]

FileNotFoundError: [Errno 2] No such file or directory: '/nvme/shared_data/llama2/huggingface/llama-2-7b-chat/../llm-awq/llama-2-7b-chat-w4-g128-awq.pt'

The command is:

python lmdeploy/serve/turbomind/deploy.py llama2 /nvme/shared_data/llama2/huggingface/llama-2-7b-chat/ --model_format awq --group_size 128 --quant_path ../llm-awq/llama-2-7b-chat-w4-g128-awq.pt --dst-path workspace/llama-2-7b-chat-w4

[lvhan028]

When compling turbomind, it raised error:
ptxas /tmp/tmpxft_000163cd_00000000-9_gemm_s4_f16.compute_70.ptx, line 80871; error : Feature 'movmatrix' requires .target sm_75 or higher
ptxas /tmp/tmpxft_000163cd_00000000-9_gemm_s4_f16.compute_70.ptx, line 80902; error : Feature 'movmatrix' requires .target sm_75 or higher
ptxas /tmp/tmpxft_000163cd_00000000-9_gemm_s4_f16.compute_70.ptx, line 80927; error : Feature 'movmatrix' requires .target sm_75 or higher
ptxas /tmp/tmpxft_000163cd_00000000-9_gemm_s4_f16.compute_70.ptx, line 80948; error : Feature 'movmatrix' requires .target sm_75 or higher
ptxas /tmp/tmpxft_000163cd_00000000-9_gemm_s4_f16.compute_70.ptx, line 80969; error : Feature 'movmatrix' requires .target sm_75 or higher

[lvhan028]

After adding -DSM=80 and -DCMAKE_CUDA_ARCHITECTURES=80, it can be built successfully

[lvhan028]

Use turbomind/chat.py to talk with AI, it is aborted

[lvhan028]

Use turbomind/client.py to communicate via TIS, it raised an error:
UNAVAILABLE: Not found: unable to load shared library: /opt/tritonserver/backends/turbomind/libtransformer-shared.so: undefined symbol: _ZN9turbomind9GemmS4F16D1Ev

[grimoire]

LGTM