Prototyping Harness Sim Testbench

[jamesdunn]

I haven't made as much progress on this as I'd like over the last couple weeks, so creating a feature request issue to keep it on my list and put down what I've thought about so far.

The feature request is to create simulation testbenches for FPGA harnesses that verify IO functionality. Someone submitting a PR for a new FPGA board could run these tests, or we could make it part of our CI.

It seems straightforward enough to use VCS to simulate Vivado-generated Verilog, including behavioral models provided for Xilinx IP blocks, since the .f files generated in the prototyping make flow include pointers to any Xilinx IP used.

I'll add a VCS sim with arguments and includes as a make target to fpga/Makefile, reusing a lot of what is already in sims/vcs/Makefile.

A simple testbench wrapper around the FPGA testharness that shorts certain Tx and Rx signals would allow a loopback test for many interfaces like UART, JTAG, and SPI. Then compiled loopback test programs like this one for UART can be ran by the simulator binary, and I can write new test programs for JTAG and SPI.

Let me know if people have any feedback.

[jamesdunn]

Update on progress: the branch arty-sim now has a config and Make SUB_PROJECT that builds with a SimSerial model, and the harness success IO not tied off.

Now I need to write an appropriate TestDriver.v to place the Harness in a configuration that tests the IOs of peripherals like UART and JTAG, and add a make target that calls VCS with proper arguments.

I'm also thinking about a concurrent XSIM testbench that would allow users without access to a VCS license to test their FPGA prototyping harnesses.

[jamesdunn]

I'm now able to generate a VCS sim binary for an ArtyTestDriver that submodules the ArtyTestHarness. This includes all of the behavioral models for any Xilinx IP used. Here's what I did to get there:

The current way that Chipyard provides behavioral models to VCS is by passing Verilog and SystemVerilog in list files with the -f flag, and the C++ collateral with includes in the CFLAGS. This works when all of the input behavioral HDL is in a single language like Verilog. All of the Xilinx IP primitives and most of the macro blocks have behavioral Verilog, and can be passed in as described here. However, there are some macro blocks where only behavioral VHDL is provided, and there is one such piece of IP used in Arty (reset_sys).

In order to do a mixed-language simulation, all of the HDL sources need to be compiled to a VCS library, using separate vlogan and vhdlan analysis tools for Verilog and VHDL, respectively. I was able to run these commands with the right flags to compile all of the Xilinx IP, the Chipyard design, and the Chipyard sim HDL collateral to a single library. Then, I just needed to pass this library to VCS for elaboration. I found that VCS does not seem to work when by compiling only some of the sources to a VCS library, then passing remaining sources as Verilog alongside the VCS library. The sources need to be all in Verilog, or all in compiled libraries. There could be a way around this, and I will reach out to Synopsys.

I did all of this by hacking some of the proprietary VCS scripts that are generated by Vivado, so I don't want to commit those. When I implement these steps in a Makefile I will commit.

However, the sim binary is not building completely. When I pass it a compiled program, it complains about not understanding this as an argument. This is because VCS does not seem to be bringing in any of the C++ collateral (for things like testchip_tsi.cc that allow programs to be passed as arguments to the sim binary) via the includes that are specified in the CFLAGS. I confirmed this by noticing that VCS does not produce any intermediate .o or .d files for this C++ collateral in its temp directory, but it does for the current Chipyard simulation make target.

I have the feeling that there is some issue with including C++ collateral in this way alongside compiled VCS source libraries, similar to how I could not include Verilog sources alongside compiled VCS source libraries; it had to be one or the other.

When I figure this out, I should have a fully-functional Arty simulator binary, able to run the standard assembly tests. Then, I will write some loopback programs for exercising the peripherals.

[jamesdunn]

OK, I've figured out how to add System C sources to the build for the mixed-language VCS flow (Synopsys calls this the "three-step analysis UUM flow"). However, even when compiling all of the System C sources to a shared library, VCS still misses the program-as-arguent functionality specified in testchip_tsi.cc, and gives the same error as above. I've gone through the flags and am still stumped as to why this is. I reached out to Synopsys with this help ticket. It's kind of long so mostly here as reference. If anyone has an idea for why the testchip_tsi functionality is being missed in this compilation, please let me know. Otherwise, I will update with what Synopsys says.

Hello, I am encountering an issue with my VCS mixed-language UUM flow. We are moving from a compile-time flow to a mixed-language UUM flow with the introduction of some VHDL IP to our design.

Previously, our design consisted entirely of Verilog, System Verilog, and System C. We invoked VCS with -f flags and list files containing the files. We also passed -CFLAGS, -LDFLAGS, and other flags specific to the design. Here is that exact command:

Here is the exact command I was using for the compile-time flow:

vcs -full64 -CFLAGS ' -O3 -std=c++11 -I/home/dunn/chipyard/riscv-tools-install/include -I/home/dunn/chipyard-dev/tools/DRAMSim2 -I/working-dir ' -LDFLAGS '-L/home/dunn/chipyard/riscv-tools-install/lib -Wl,--no-as-needed,-rpath,/home/dunn/chipyard/riscv-tools-install/lib -L/home/dunn/chipyard-dev/fpga -L/home/dunn/chipyard-dev/tools/DRAMSim2' -lfesvr -ldramsim -notice -line +lint=all,noVCDE,noONGS,noUI -error=PCWM-L -error=noZMMCM -timescale=1ns/10ps -quiet -q +rad +vcs+lic+wait +vc+list \
-f /working-dir/sim_files.common.f \
-y $XILINX_VIVADO/data/verilog/src/unisims \
-y $XILINX_VIVADO/data/verilog/src/xeclib \
+incdir+$XILINX_VIVADO/data/verilog/src +libext+.v \
$XILINX_VIVADO/data/verilog/src/glbl.v -sverilog +systemverilogext+.sv+.svi+.svh+.svt -assert svaext +libext+.sv +v2k +verilog2001ext+.v95+.vt+.vp +libext+.v -debug_pp \
+incdir+/working-dir /working-dir/design.top.v /working-dir/design.harness.v /working-dir/design.top.mems.v /working-dir/design.harness.mems.v +define+VCS +define+CLOCK_PERIOD=1.0 +define+RESET_DELAY=777.7 +define+PRINTF_COND='TestDriver.printf_cond' +define+STOP_COND='!TestDriver.reset' +define+RANDOMIZE_MEM_INIT +define+RANDOMIZE_REG_INIT +define+RANDOMIZE_GARBAGE_ASSIGN +define+RANDOMIZE_INVALID_ASSIGN  -o /home/dunn/chipyard-dev/fpga/simv-chipyard-TinyRocketArtySimConfig -Mdir=/working-dir/design

Now, I am using the following vhdlan, vlogan, and syscan commands as part of the three-step UUM flow:

vhdlan -full64 -work xil_defaultlib  \
"/working-dir/obj/ip/reset_sys/sim/reset_sys.vhd"

vlogan -full64 +v2k -work xil_defaultlib +define+VCS +define+CLOCK_PERIOD=1.0 +define+RESET_DELAY=777.7 +define+PRINTF_COND='ArtyTestDriver.printf_cond' +define+STOP_COND='!ArtyTestDriver.reset' +define+RANDOMIZE_MEM_INIT +define+RANDOMIZE_REG_INIT +define+RANDOMIZE_GARBAGE_ASSIGN +define+RANDOMIZE_INVALID_ASSIGN +incdir+/working-dir/obj/ip/mmcm \
"/working-dir/obj/ip/mmcm/mmcm_clk_wiz.v" \
"/working-dir/obj/ip/mmcm/mmcm.v" \
"/working-dir/EICG_wrapper.v" \
"/working-dir/IOCell.v" \
"/working-dir/../../fpga-shells/xilinx/common/vsrc/PowerOnResetFPGAOnly.v" \
"/working-dir/design.harness.mems.v" \
"/working-dir/design.harness.v" \
"/working-dir/design.top.mems.v" \
"/working-dir/design.top.v" \
"/working-dir/plusarg_reader.v"

vlogan -full64 -sverilog -Xcheck_p1800_2009=char -work xil_defaultlib +define+VCS +define+CLOCK_PERIOD=1.0 +define+RESET_DELAY=777.7 +define+PRINTF_COND='ArtyTestDriver.printf_cond' +define+STOP_COND='!ArtyTestDriver.reset' +define+RANDOMIZE_MEM_INIT +define+RANDOMIZE_REG_INIT +define+RANDOMIZE_GARBAGE_ASSIGN +define+RANDOMIZE_INVALID_ASSIGN +incdir+/working-dir/obj/ip/mmcm \
"/working-dir/ClockDividerN.sv" \
"/working-dir/ArtyTestDriver.v" \
"/working-dir/SimSerial.v"

vlogan -full64 +v2k -work xil_defaultlib "/working-dir/glbl.v"

syscan -V -full64 -work xil_defaultlib -cpp '/home/dunn/GCC-7.3.0/bin/g++-7.3' -CFLAGS '-O3 -std=c++11 -I/home/dunn/chipyard/riscv-tools-install/include -I/home/dunn/chipyard-dev/tools/DRAMSim2 -I/working-dir' -f /working-dir/sim_cc_files.f -Mdir=/working-dir/design

This should compile all of the sources to the VCS work library "xil_defaultlib". I have mapped this library to a directory using a synopsys_sim.setup file.

I then run VCS like so:

vcs -cpp '/home/dunn/GCC-7.3.0/bin/g++-7.3' -sysc -full64 -CFLAGS '-O3 -std=c++11 -I/home/dunn/chipyard/riscv-tools-install/include -I/home/dunn/chipyard-dev/tools/DRAMSim2 -I/working-dir' -LDFLAGS '-L/home/dunn/chipyard/riscv-tools-install/lib -Wl,--no-as-needed,-rpath,/home/dunn/chipyard/riscv-tools-install/lib -L/home/dunn/chipyard-dev/fpga -L/home/dunn/chipyard-dev/tools/DRAMSim2' -lfesvr -ldramsim -notice -line +lint=all,noVCDE,noONGS,noUI -error=PCWM-L -error=noZMMCM -timescale=1ns/10ps -quiet -q +rad +vcs+lic+wait +vc+list -debug_pp xil_defaultlib.ArtyTestDriver xil_defaultlib.glbl -o /working-dir/simv-chipyard-TinyRocketArtySimConfig -Mdir=/working-dir/design

The issue that I run into is that these System C files contain code that should allow the VCS simulation binary to accept arguents, particularly a program binary to run as its argument like so:

/working-dir/simv-chipyard-TinyRocketArtySimConfig some-binary

This works for the compile-time flow, and the VCS simulation binary takes a program to run as its argument.

However, after compiling the VCS binary with the three-step UUM flow, it does not accept the program as an argument, and complains:

Warning-[RUNT-OPT-UNKNOWN] Unknown runtime option
  Unknown VCS runtime option
  'some-binary'
  passed as part of command './simv-chipyard-TinyRocketArtySimConfig some-binary
  Please review valid vcs runtime options by executing ./<simulation
  executable> -help

I also tried including just the C files at compile time using the -file flag, like this:

vcs -full64 -CFLAGS '-O3 -std=c++11 -I/home/dunn/chipyard/riscv-tools-install/include -I/home/dunn/chipyard-dev/tools/DRAMSim2 -I/working-dir' -LDFLAGS '-L/home/dunn/chipyard/riscv-tools-install/lib -Wl,--no-as-needed,-rpath,/home/dunn/chipyard/riscv-tools-install/lib -L/home/dunn/chipyard-dev/fpga -L/home/dunn/chipyard-dev/tools/DRAMSim2' -lfesvr -ldramsim -notice -line +lint=all,noVCDE,noONGS,noUI -error=PCWM-L -error=noZMMCM -timescale=1ns/10ps -quiet -q +rad +vcs+lic+wait +vc+list -lca -sverilog -assert svaext -debug_pp xil_defaultlib.ArtyTestDriver xil_defaultlib.glbl -file /working-dir/sim_cc_files.f -o /working-dir/simv-chipyard-TinyRocketArtySimConfig -Mdir=/working-dir/design

The stdout looks pretty much identical to the compile-time flow, saying that all of the System C sources were compiled with g++. And there are intermediate compilation files for the System C sources in the -Mdir. However, I still cannot pass any arguments to the VCS simulation binary.

Is there something with the System C files that I am missing in my UUM flow? Any flags? Thank you.

[jamesdunn]

Figured out why I am not able to pass the binary as an argument to the sim binary generated in this flow.

This reset condition in SimSerial.v is never exited, so all of the RTL outside of the reset condition is likely pruned. This means serial_tick, which calls testchip_tsi_t, which takes argc/argvs, is not included, which is why the program is not recognized as an argument to the simulation binary.

As for why we don't leave the reset condition, my suspicion is that it is related to the behavioral model for the Xilinx reset state machine that is included in the design.

Will generate waveforms to see what is going on. Closing this issue, and the discussion can continue in #871.