MongoDB uses zstd and QATzip

[Microcarft]

I would like to know if, when using MongoDB and switching to the zstd compression algorithm, I can use this plugin to call QAT hardware?

[cfzhux]

Yes, MongoDB can use qat to accelerate the compression

[Microcarft]

是的，MongoDB 可以使用 qat 来加速压缩

Thank you very much for your answer. I'm curious whether I should use QATlib+QATzip or directly use QAT-ZSTD-Plugin after installing the driver.

[cfzhux]

你是想使用QAT加速mongdb中的ZSTD吗？如果是的，不用安装qatzip，需要安装qatlib和qat zstd plugin, 然后参考https://github.com/intel/QAT-ZSTD-Plugin?tab=readme-ov-file#how-to-integrate-qat-sequence-producer-into-applications 这部分添加一些代码

[Microcarft]

Thanks, and I would also like to ask if I need to install both Intel® QAT Driver for Linux* Hardware v2.0 and QATlib? I am using the latest 6.9.7 kernel for testing and I am not sure what will happen next.

[cfzhux]

Thanks, and I would also like to ask if I need to install both Intel® QAT Driver for Linux* Hardware v2.0 and QATlib? I am using the latest 6.9.7 kernel for testing and I am not sure what will happen next.

it seems that you are using QAT OOT package, you can refer to https://intel.github.io/quickassist/GSG/2.X/installation.html, you don't need to install QATlib(https://github.com/intel/qatlib), it is for QAT In-tree software package

[Microcarft]

谢谢，我还想问一下我是否需要同时安装英特尔® QAT 驱动程序（适用于 Linux* 硬件 v2.0）和 QATlib？我正在使用最新的 6.9.7 内核进行测试，但不确定接下来会发生什么。

看来你正在使用 QAT OOT 包，你可以参考https://intel.github.io/quickassist/GSG/2.X/installation.html，你不需要安装 QATlib( https://github.com/intel/qatlib )，它是用于 QAT In-tree 软件包

Oh, that's really unfortunate. After installing the qat_4xxx module in the kernel, I tried to install it using the package mentioned in the link. I encountered the following error when making:
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_transport_debug.c: In function ‘adf_ring_debugfs_add’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_transport_debug.c:117:9: error: implicit declaration of function ‘strlcpy’; did you mean ‘strncpy’? [-Werror=implicit -function-declaration] 117 | strlcpy(ring_debug->ring_name, name, sizeof(ring_debug->ring_name)); | ^~~~~~~ | strncpy /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_pasid.c:184:10: error: ‘const s truct mmu_notifier_ops’ has no member named ‘invalidate_range’; did you mean ‘invalida te_range_end’? 184 | .invalidate_range = adf_dev_mm_invalidate_range, | ^~~~~~~~~~~~~~~~ | invalidate_range_end /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_pasid.c:184:29: error: initiali zation of ‘int (*)(struct mmu_notifier *, struct mm_struct *, long unsigned int, long unsigned int)’ from incompatible pointer type ‘void (*)(struct mmu_notifier *, struct mm_struct *, long unsigned int, long unsigned int)’ [-Werror=incompatible-pointer-ty pes] 184 | .invalidate_range = adf_dev_mm_invalidate_range, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~ /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_pasid.c:184:29: note: (near ini tialization for ‘adf_dev_mmu_notifier_ops.clear_flush_young’) /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_bundle.c: In function ‘adf_ cfg_get_ring_pairs_from_bundle’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_bundle.c:100:9: error: impl icit declaration of function ‘strlcpy’; did you mean ‘strncpy’? [-Werror=implicit-func tion-declaration] 100 | strlcpy(bundle->sections[bundle->section_index], | ^~~~~~~ | strncpy /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_ctl_drv.c: In function ‘adf_get _dev_node_id’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_ctl_drv.c:551:22: error: ‘struc t cpuinfo_x86’ has no member named ‘phys_proc_id’ 551 | if (c->phys_proc_id == 0) { | ^~ /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_ctl_drv.c:554:55: error: ‘struc t cpuinfo_x86’ has no member named ‘phys_proc_id’ 554 | bus_per_cpu = MAX_PCI_BUS / (c->phys_proc_id + 1); | ^~ /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_fw_counters.c: In function ‘adf _fw_counters_add_key_value_param’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_fw_counters.c:347:9: error: imp licit declaration of function ‘strlcpy’; did you mean ‘strncpy’? [-Werror=implicit-fun ction-declaration] 347 | strlcpy(key_val->key, key, sizeof(key_val->key)); | ^~~~~~~ | strncpy /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_ctl_drv.c: In function ‘adf_ctl _ioctl_get_status’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_ctl_drv.c:587:9: error: implici t declaration of function ‘strlcpy’; did you mean ‘strncpy’? [-Werror=implicit-functio n-declaration] 587 | strlcpy(dev_info.name, hw_data->dev_class->name, sizeof(dev_info.name) ); | ^~~~~~~ | strncpy cc1: some warnings being treated as errors /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_device.c: In function ‘adf_ cfg_get_user_section’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_device.c:191:25: error: imp licit declaration of function ‘strlcpy’; did you mean ‘strncpy’? [-Werror=implicit-fun ction-declaration] 191 | strlcpy(user_sec, | ^~~~~~~ | strncpy make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_pasid.o] Error 1 make[6]: *** Waiting for unfinished jobs.... cc1: some warnings being treated as errors /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg.c: In function ‘adf_cfg_add _key_value_param’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg.c:271:9: error: implicit de claration of function ‘strlcpy’; did you mean ‘strncpy’? [-Werror=implicit-function-de claration] 271 | strlcpy(key_val->key, key, sizeof(key_val->key)); | ^~~~~~~ | strncpy LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_d15xxvf/qat_d15xxvf.o make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_ctl_drv.o] Error 1 LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_c3xxx/qat_c3xxx.o LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_c62x/qat_c62x.o LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_d15xx/qat_d15xx.o LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_c4xxx/qat_c4xxx.o LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.o LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_dh895xccvf/qat_dh895xccvf.o /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_aer.c: In function ‘adf_enable_ aer’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_aer.c:381:15: error: implicit d eclaration of function ‘pci_enable_pcie_error_reporting’ [-Werror=implicit-function-de claration] 381 | ret = pci_enable_pcie_error_reporting(pdev); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_c3xxxvf/qat_c3xxxvf.o /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_section.c: In function ‘adf _cfg_set_core_number_for_instance’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_section.c:73:17: error: imp licit declaration of function ‘strlcpy’; did you mean ‘strncpy’? [-Werror=implicit-fun ction-declaration] 73 | strlcpy(tokens[token_index++], core_val, | ^~~~~~~ | strncpy LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_c62xvf/qat_c62xvf.o cc1: some warnings being treated as errors make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_transport_debug.o] Error 1 /QAT/quickassist/qat/drivers/crypto/qat/qat_4xxx/adf_4xxx_pm.c: In function ‘get_cfg_p m_setting’: /QAT/quickassist/qat/drivers/crypto/qat/qat_4xxx/adf_4xxx_pm.c:21:9: error: implicit d eclaration of function ‘strlcpy’; did you mean ‘strncpy’? [-Werror=implicit-function-d eclaration] 21 | strlcpy(key, key_str, sizeof(key)); | ^~~~~~~ | strncpy /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_adi.c: In function ‘adf_init_ad is’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_adi.c:145:9: error: implicit de claration of function ‘strlcpy’; did you mean ‘strncpy’? [-Werror=implicit-function-de claration] 145 | strlcpy(key, ADF_NUM_ADI, ADF_CFG_MAX_VAL_LEN_IN_BYTES); | ^~~~~~~ | strncpy LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_c4xxxvf/qat_c4xxxvf.o /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_aer.c: In function ‘adf_disable _aer’: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_aer.c:404:9: error: implicit de claration of function ‘pci_disable_pcie_error_reporting’ [-Werror=implicit-function-de claration] 404 | pci_disable_pcie_error_reporting(pdev); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ cc1: some warnings being treated as errors make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_fw_counters.o] Error 1 LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_4xxxvf/qat_4xxxvf.o cc1: some warnings being treated as errors make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_bundle.o] Error 1 cc1: some warnings being treated as errors cc1: some warnings being treated as errors make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg.o] Error 1 make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_aer.o] Error 1 cc1: some warnings being treated as errors make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_adi.o] Error 1 cc1: some warnings being treated as errors make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_4xxx/adf_4xxx_pm.o] Error 1 make[6]: *** Waiting for unfinished jobs.... LD [M] /QAT/quickassist/qat/drivers/crypto/qat/qat_vqat/qat_vqat.o make[5]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:490: /QAT /quickassist/qat/drivers/crypto/qat/qat_4xxx] Error 2 make[5]: *** Waiting for unfinished jobs.... cc1: some warnings being treated as errors make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_device.o] Error 1 cc1: some warnings being treated as errors make[6]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:249: /QAT /quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_section.o] Error 1 make[5]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:490: /QAT /quickassist/qat/drivers/crypto/qat/qat_common] Error 2 make[4]: *** [/usr/src/linux-headers-6.9.7+bpo-common/scripts/Makefile.build:490: /QAT /quickassist/qat/drivers/crypto/qat] Error 2 make[3]: *** [/usr/src/linux-headers-6.9.7+bpo-common/Makefile:1943: /QAT/quickassist/ qat] Error 2 make[2]: *** [/usr/src/linux-headers-6.9.7+bpo-common/Makefile:252: __sub-make] Error 2 make[2]: Leaving directory '/usr/src/linux-headers-6.9.7+bpo-amd64' make[1]: *** [Makefile:69: modules] Error 2 make[1]: Leaving directory '/QAT/quickassist/qat' make: *** [Makefile:1398: qat-driver-all] Error 2 root@example:/QAT# lsmod | grep qat qat_4xxx 20480 0 intel_qat 364544 1 qat_4xxx crc8 12288 1 intel_qat authenc 12288 1 intel_qat
So I had to try another option, which is QATlib, and it has been successfully run:
`● qat.service - QAT service
Loaded: loaded (/lib/systemd/system/qat.service; enabled; preset: enabled)
Active: active (running) since Mon 2024-08-26 12:18:12 CST; 2h 1min ago
Main PID: 27883 (qatmgr)
Tasks: 1 (limit: 629145)
Memory: 892.0K
CPU: 2.673s
CGroup: /system.slice/qat.service
└─27883 /usr/sbin/qatmgr --policy=0

Aug 26 12:18:08 example[1]: Starting qat.service - QAT service...
Aug 26 12:18:12 example[1]: Started qat.service - QAT service.
`

[Microcarft]

谢谢，我还想问一下是否需要同时安装英特尔® QAT 驱动程序（适用于 Linux* 硬件 v2.0）和 QATlib？我正在使用最新的 6.9.7 内核进行测试，但不确定接下来会发生什么。

看来你正在使用QAT OOT包，你可以参考https://intel.github.io/quickassist/GSG/2.X/installation.html，你不需要安装QATlib( https://github.com/intel/ qatlib )，它是用于 QAT In-tree 的

Sorry, the problem with ZSTD-Plugin has not been solved yet. I tried to find the three APIs you mentioned in Mongo, but there was no result. It seems that Mongo's WiredTiger does not call these three APIs?

[cfzhux]

谢谢，我还想问一下是否需要同时安装英特尔® QAT 驱动程序（适用于 Linux* 硬件 v2.0）和 QATlib？我正在使用最新的 6.9.7 内核进行测试，但不确定接下来会发生什么。

看来你正在使用QAT OOT包，你可以参考[https://intel.github.io/quickassist/GSG/2.X/installation.html，你不需要安装QATlib(](https://intel.github.io/quickassist/GSG/2.X/installation.html%EF%BC%8C%E4%BD%A0%E4%B8%8D%E9%9C%80%E8%A6%81%E5%AE%89%E8%A3%85QATlib() https://github.com/intel/ qatlib )，它是用于 QAT In-tree 的

Sorry, the problem with ZSTD-Plugin has not been solved yet. I tried to find the three APIs you mentioned in Mongo, but there was no result. It seems that Mongo's WiredTiger does not call these three APIs?

Could you try to use ZSTD_compress2 or ZSTD_compressStream2 to replace the API called by Mongo?

[Microcarft]

谢谢，我还想问一下是否需要同时安装英特尔® QAT 驱动程序（适用于 Linux* 硬件 v2.0）和 QATlib？我正在使用最新的 6.9.7 内核进行测试，但不确定接下来会发生什么。

看来你正在使用QAT OOT包，你可以参考[https://intel.github.io/quickassist/GSG/2.X/installation.html，你不需要安装QATlib(](https://intel.github.io/quickassist/GSG/2.X/installation.html%EF%BC%8C%E4%BD%A0%E4%B8%8D%E9%9C%80%E8%A6%81%E5%AE%89%E8%A3%85QATlib() https://github.com/intel/ qatlib )，它是用于 QAT In-tree 的

Sorry, the problem with ZSTD-Plugin has not been solved yet. I tried to find the three APIs you mentioned in Mongo, but there was no result. It seems that Mongo's WiredTiger does not call these three APIs?

Could you try to use ZSTD_compress2 or ZSTD_compressStream2 to replace the API called by Mongo?

Thanks for your reply!
I am trying it as you said.
I successfully compiled QAT-ZSTD-Plugin not long ago and got the libqatseqprod.so file. Then I found the source code of zstd1.5.5 in the source code of MongoDB and modified the code according to this instruction:

How to integrate QAT sequence producer into zstd

Then the zstd module was successfully compiled and run.

However, when I performed a simple performance test on the zstd module with QAT-ZSTD-Plugin added, I found that the compression time of multiple test samples was much higher than that of the normal zstd module compiled with the same source code. This is very puzzling, but I still continue to try to compile MongoDB directly to verify whether this set of processes can succeed - even if the performance of the zstd module with QAT is not normal.

[cfzhux]

for the performance, you can refer to https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Intel-QuickAssist-Technology-Zstandard-Plugin-an-External/post/1509818