Error Building Static Application using CMake (U250)

[aaqdas]

I'm getting an error message when building static application for u250. I run the following command

cmake ../CMakeLists.txt -DFDEV_NAME=u250 -DEXAMPLE=static

Error Message

-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Vivado at /home/celeris/tools/Xilinx_2022.1/Vivado/2022.1.
-- Found Vivado/Vitis HLS at /home/celeris/tools/Xilinx_2022.1/Vitis_HLS/2022.1.
** Vitis toolchain
** Static design flow
** Target platform u250
CMake Error at /home/celeris/Documents/aqdas/coyote-v2/Coyote/cmake/FindCoyoteHW.cmake:398 (message):
  Number of DDR channels misconfigured.
Call Stack (most recent call first):
  CMakeLists.txt:31 (validation_checks_hw)


-- Configuring incomplete, errors occurred!
See also "/home/celeris/Documents/aqdas/coyote-v2/Coyote/examples_hw/CMakeFiles/CMakeOutput.log".

Can you please look into the issue and update on this?

[aaqdas]

Another problem I face is that, once I have run CMake for one application, and I try to run CMake again for another application, an error is thrown

CMake Error at /home/celeris/Documents/aqdas/Coyote/cmake/FindCoyoteHW.cmake:555 (cmake_parse_arguments):
  Unknown CMake command "cmake_parse_arguments".
Call Stack (most recent call first):
  CMakeLists.txt:197 (load_apps)

[maximilianheer]

Hi! Regarding your first request, I looked into the cmake and it seems as if the N_DDR_CHAN-variable is not properly set for DDR-based platforms - I tried to address that in the latest commit, so you might want to give a try. However: We have decided to stop the support for any non-HBM platforms and will from now on only continue the development and support for the Alveo U55C accelerator card (see updated Readme). Thanks a lot for your understanding!

For your second request, I'd be happy if you could provide the full context (essentially your terminal-input) to get a better understanding of the issue.

Thanks for your feedback through requests - always appreciated!

Kind regards,

Max.

[aaqdas]

Hi,

Thank you for your support, Max. Do you know which commit had design that was fully tested on U-250?

I resolved the second problem by using /usr/bin/cmake instead of cmake. It was a problem with the CMake that comes with Vivado, which was set as default in my terminal. It didn't include the package for cmake_parse_arguments by default.

[maximilianheer]

Hi!

I don't think that Coyote v2 was ever tested on U250. So I guess the last version that might work on this platform out of the box would be Coyote v1 (see branch "coyote_v1"). Depending on what you're trying to achieve, the functionality of this shell should be very similar, but the architecture is actually quite different.

Let me know if that helps you.

Best,

Max.

[aaqdas]

Thank you very much for your support, once again. I'll look into the first version.

[aaqdas]

I'm trying v2 at the moment, and it works for most of the part. I tried the static-shell with perf-local application and it works great. I actually wanted to test out the RDMA.

I get the static_routed_locked.dcp checkpoint in the correct directory i.e. /hw/static_routed_locked_u250.dcp. Then I run the synthesis of rdma-perf hardware which completes the entire workflow of make bitgen.

However, it doesn't show enabled RDMA in dmesg logs.

So, I tried my own custom CMake Script to Build What I Want, An RDMA Enabled Static Shell with Performance Tester.

if(EXAMPLE STREQUAL "static_rdma")
    set(BUILD_STATIC 1)
    set(BUILD_SHELL 0)
    set(BUILD_OPT 1)
    set(N_REGIONS 4)
    set(EN_STRM 1)
    set(EN_MEM 1)
    set(HBM_SPLIT 1)
    set(EN_RDMA 1)
    set(SHELL_PROBE 6)
    set(N_STRM_AXI 2)
    
    validation_checks_hw()

    load_apps (
        VFPGA_C0_0 "apps/rdma_perf"
        VFPGA_C0_1 "apps/perf_local"
        VFPGA_C0_2 "apps/perf_local"
        VFPGA_C0_3 "apps/perf_local"
    )

    create_hw()
endif()

Its configured properly and goes well until make bitgen where the following error is thrown in RDMA Implementation.

ERROR: [Synth 8-660] unable to resolve 'request_snk' [/home/celeris/Documents/aqdas/Coyote_V2/Coyote/hw/hdl/network/rdma/rdma_meta_tx_arbiter.sv:163]
ERROR: [Synth 8-6156] failed synthesizing module 'rdma_meta_tx_arbiter' [/home/celeris/Documents/aqdas/Coyote_V2/Coyote/hw/hdl/network/rdma/rdma_meta_tx_arbiter.sv:37]
ERROR: [Synth 8-6156] failed synthesizing module 'rdma_arbiter' [/home/celeris/Documents/aqdas/Coyote_V2/Coyote/hw/hdl/network/rdma/rdma_arbiter.sv:40]
ERROR: [Synth 8-6156] failed synthesizing module 'design_dynamic_top' [/home/celeris/Documents/aqdas/Coyote_V2/Coyote/examples_hw/build_static_rdma/test_shell/hdl/dynamic_top.sv:35]
ERROR: [Synth 8-6156] failed synthesizing module 'shell_top' [/home/celeris/Documents/aqdas/Coyote_V2/Coyote/examples_hw/build_static_rdma/test_shell/hdl/shell_top.sv:35]
---------------------------------------------------------------------------------
Finished RTL Elaboration : Time (s): cpu = 00:00:07 ; elapsed = 00:00:08 . Memory (MB): peak = 3715.883 ; gain = 489.578 ; free physical = 36878 ; free virtual = 56685
---------------------------------------------------------------------------------
RTL Elaboration failed
INFO: [Common 17-83] Releasing license: Synthesis
218 Infos, 206 Warnings, 0 Critical Warnings and 6 Errors encountered.
synth_design failed
ERROR: [Common 17-69] Command failed: Synthesis failed - please see the console or run log file for details
INFO: [Common 17-206] Exiting Vivado at Thu Sep 12 20:35:01 2024...
[Thu Sep 12 20:35:12 2024] synth_1 finished
WARNING: [Vivado 12-8222] Failed run(s) : 'synth_1'
wait_on_runs: Time (s): cpu = 02:16:04 ; elapsed = 00:21:11 . Memory (MB): peak = 6884.355 ; gain = 0.000 ; free physical = 39337 ; free virtual = 59144
** CERR: ERROR: [Common 17-69] Command failed: Run 'synth_1' failed. Unable to open

**
INFO: [Common 17-206] Exiting Vivado at Thu Sep 12 20:35:13 2024...
make[3]: *** [CMakeFiles/bitgen.dir/build.make:89: checkpoints/shell/shell_synthed.dcp] Error 1
make[2]: *** [CMakeFiles/Makefile2:269: CMakeFiles/bitgen.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:276: CMakeFiles/bitgen.dir/rule] Error 2
make: *** [Makefile:118: bitgen] Error 2

I believe the following errors are also caused by the first error. For reference, I'm working with Vivado 2022.1 as specified in the documentation.

[aaqdas]

I believe it may be a typo, since there's only one use of request_snk and there's extensive declaration and use of req_snk.

[aaqdas]

Using Coyote-V1, I was able to communicate between two Coyote RDMAs. However, it gets stuck after transmitting a few packets. I've attached a screenshot below

Here it shows that Coyote Slave Transmitted (209) Packets and Coyote Master Responded with (1) Packet. However, it doesn't go any further than this.

I'd highly appreciate your support with this.

[maximilianheer]

Hi @aaqdas, I'm sorry that you're facing issues with coyote-v1. We are indeed aware that RDMA has some stability issues, especially when used via a switched network. In our experience, it has been useful to a) directly plug the FPGAs into each other to avoid all switch-interactions (some switches are actually kinda sensitive to RDMA-traffic) and b) use the pcap-capability in the switch to sniff the FPGA-outgoing traffic to see which entity is struggling with retransmissions. After that, you might want to install pcaps in the RDMA-networking stack of this entitiy to check the interfaces for potential problems. Coyote v1 - which you seem to be using - has some flaws when handling retransmissions that were only solved during the migration to v2. We do not plan to retrofit v1 in this regard.

[aaqdas]

Hi,

Thank you for your support. I was able to generate traffic and receive it on the other end.

[bo3z]

Hi @aaqdas to get it to work, did you have any fixes / code changes in Coyote? If so, would you mind opening a PR or pushing your code on a branch. We really appreciate open-source contributions.

[aaqdas]

Hi @bo3z

We went through a series of iterations. We were able to run the design from coyote-v1 branch. The design was synthesized using the following commands.

cd hw && mkdir build
cd build
cmake ../ -DEXAMPLE=perf_rdma_host -DFDEV_NAME=u250

We had to use perf_rdma_host. Previously, we had been trying perf_rdma_card. We're still unable to figure out how to retrieve data from the card memory to host memory. We tried invoke operations used in perf_ddr and perf_hosts, with theSYNC and OFFLOAD Operations.

We used coyote on both the client and server hosts.

On the software side, we had to use the same operation on both the server and client applications (either read or write). We also had to set the huge pages beforehand.