[LLM Runtime] refactor itrex backend based on the latest Jblas

[zhewang1-intc]

Type of Change

feature or bug fix or documentation or others
API changed or not

Description

detail description
JIRA ticket: xxx

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

[DDEle]

Glad to see the great refactor!

(seem someone need to fix the cpp graph)

[luoyu-intel]

Next-token beam_number=1 Xeon8480+ 48 threads
group_size=128
CompType=fp32 1.3x speedup
main branch:
perf_total_per_op_us[ FFN_SILU] = 16.995 ms
perf_total_per_op_us[ INNER PRODUCT] = 13.321 ms

pr branch:
perf_total_per_op_us[ MUL_QKV] = 5.460 ms
perf_total_per_op_us[ FFN_SILU] = 12.781 ms
perf_total_per_op_us[ INNER PRODUCT] = 2.931 ms

CompType=int8 1.08x
main:
perf_total_per_op_us[ MUL_QKV] = 4.812 ms
perf_total_per_op_us[ FFN_SILU] = 11.572 ms
perf_total_per_op_us[ INNER PRODUCT] = 2.455 ms

pr branch:
perf_total_per_op_us[ MUL_QKV] = 4.343 ms
perf_total_per_op_us[ FFN_SILU] = 10.654 ms
perf_total_per_op_us[ INNER PRODUCT] = 2.180 ms

comp_type=bf16, ~1.3x
main(no ffn, qkv fusion) :
perf_total_per_op_us[ INNER PRODUCT] = 41.611 ms

pr branch:
perf_total_per_op_us[ MUL_QKV] = 6.667 ms
perf_total_per_op_us[ FFN_SILU] = 16.416 ms
perf_total_per_op_us[ INNER PRODUCT] = 3.089 ms

[DDEle]

Fused-Attention part (intel_extension_for_transformers/llm/runtime/graph/core/layers/mha_dense.cpp) is ready for review.

[luoyu-intel]

long prompt len=2023 Xeon8480+ 48 threads
group_size=128

CompType=int8, 1.2x:
main:
model_print_timings: prompt eval time = 1430.17 ms / 2023 tokens ( 0.71 ms per token)
model_print_timings: eval time = 410.02 ms / 15 runs ( 27.33 ms per token)
model_print_timings: total time = 2969.41 ms
========== eval time log of each prediction ==========
prediction 0, time: 1430.17ms
prediction 1, time: 28.56ms
prediction 2, time: 27.65ms
prediction 3, time: 27.34ms
prediction 4, time: 27.46ms

pr branch:
model_print_timings: prompt eval time = 1184.35 ms / 2023 tokens ( 0.59 ms per token)
model_print_timings: eval time = 389.48 ms / 15 runs ( 25.97 ms per token)
model_print_timings: total time = 2577.57 ms
========== eval time log of each prediction ==========
prediction 0, time: 1184.35ms
prediction 1, time: 27.49ms
prediction 2, time: 26.47ms
prediction 3, time: 25.85ms
prediction 4, time: 25.77ms

CompType=bf16 1.14x,
main:
model_print_timings: prompt eval time = 1624.27 ms / 2023 tokens ( 0.80 ms per token)
model_print_timings: eval time = 722.67 ms / 15 runs ( 48.18 ms per token)
model_print_timings: total time = 3281.25 ms
========== eval time log of each prediction ==========
prediction 0, time: 1624.27ms
prediction 1, time: 49.15ms
prediction 2, time: 48.79ms
prediction 3, time: 48.60ms
prediction 4, time: 48.33ms
prediction 5, time: 48.33ms

pr branch:
model_print_timings: prompt eval time = 1422.43 ms / 2023 tokens ( 0.70 ms per token)
model_print_timings: eval time = 549.44 ms / 15 runs ( 36.63 ms per token)
model_print_timings: total time = 3063.09 ms
========== eval time log of each prediction ==========
prediction 0, time: 1422.43ms
prediction 1, time: 37.75ms
prediction 2, time: 36.69ms
prediction 3, time: 36.61ms
prediction 4, time: 36.57ms

CompType=fp32 0.98x
main:
model_print_timings: prompt eval time = 5066.65 ms / 2023 tokens ( 2.50 ms per token)
model_print_timings: eval time = 567.13 ms / 15 runs ( 37.81 ms per token)
model_print_timings: total time = 6633.77 ms
========== eval time log of each prediction ==========
prediction 0, time: 5066.65ms
prediction 1, time: 38.81ms
prediction 2, time: 38.33ms
prediction 3, time: 38.15ms
prediction 4, time: 37.60ms

pr branch:
model_print_timings: prompt eval time = 5184.93 ms / 2023 tokens ( 2.56 ms per token)
model_print_timings: eval time = 427.62 ms / 15 runs ( 28.51 ms per token)
model_print_timings: total time = 6701.66 ms
========== eval time log of each prediction ==========
prediction 0, time: 5184.93ms
prediction 1, time: 29.46ms
prediction 2, time: 29.46ms
prediction 3, time: 29.28ms
prediction 4, time: 29.23ms

[airMeng]

shall there be any updated on jblas doc https://github.com/intel/intel-extension-for-transformers/blob/refactor_itrex_backend_based_on_new_jblas/intel_extension_for_transformers/llm/library/jblas/README.md ?

@luoyu-intel @zhewang1-intc

[VincyZhang]

https://inteltf-jenk.sh.intel.com/view/nlp-toolkit-validation/job/ITREX-cpp-graph-extension/200/

[VincyZhang]

https://inteltf-jenk.sh.intel.com/job/ITREX-cpp-graph-extension/202/

[VincyZhang]

https://inteltf-jenk.sh.intel.com/view/nlp-toolkit-validation/job/ITREX-cpp-graph-extension/200/

Some model failed without output, just hangs for days (not due to disk), please check @zhewang1-intc @luoyu-intel

[VincyZhang]

https://inteltf-jenk.sh.intel.com/job/ITREX-cpp-graph-extension/203/

[VincyZhang]

https://inteltf-jenk.sh.intel.com/view/nlp-toolkit-validation/job/ITREX-cpp-graph-extension/204/

[VincyZhang]

Neuralchat UT failure is irrelevant, please check @lvliang-intel