Multi rdma device not working in rdma brpc branch

[tbago]

We have multi rdma NIC in server. If we use the first rdma device by use --rdma_device=mlx5_0. The brpc client and server works well. But if we use other rdma device like --rdma_device=mlx5_1, the client cannot communicate to the server. It will show the following error:

The rdma_create_qp will failed with errno=22 (invalid param). The root cause is that the GetRdmaProtectionDomain() is use the global context, I mean that the global context may support support well or have some bugs? I have write my custom client and use rdma_cm_id->verbs as context. Then I can connect the brpc server with other rdma device.
But in the rdma_helper.cpp the pd is init before the rdma_cm_id.
So my question is does someone have the same issue will me. And how to fixed the bug, do we need the create the pd after the rdma_cm_id is created.

[tbago]

I found the rdma_cm_id is not matched when use the second device. I have add the bind function in client side. And it can fixed the client connection isuse.

[ziruiliu]

#1183
这个描述的问题应该和你类似，目前的代码不支持同时使用其它网卡或多块网卡

[wwbmmm]

brpc has no plan to support multi rdma device.

[changchengx]

@wwbmmm
I tested the RoCE LAG device with below command and it could work

[307 /brpc/example/rdma_performance/]$ ./perf_server --rdma_device mlx5_bond_1
[306 /brpc/example/rdma_performance/]$ ./perf_client --rdma_device mlx5_bond_1 --servers=192.168.30.7:8002 --attachment_size=1024 --thread_num=10

According to the experiment's result, brpc doesn't support multi rdma devices, so it need to specify which device should be used with the option --rdma_device.

brpc supports RoCE LAG device, right?

[Tuvie]

@changchengx
Yes, brpc can support RoCE LAG device. But brpc does not support multi rdma devices without LAG.

[changchengx]

@Tuvie
Got it. Thanks