[VTA][TSIM][Build] Towards TSIM CI testing

[tmoreau89]

Before both build and execution would be controlled by the TARGET field in the vta_config.json. Now we separate the build (set in cmake with the USE_TSIM field) from the execution (set by the TARGET field in the vta_config.json.

This change presents the following advantages:

• Quickly switch between tsim and sim without having to build
• Makes colab demos more streamlined since we can build TSIM for all demo packages and control which simulator to use
• This will also enable TSIM testing in the CI (needs follow up in separate PR)

Update: now that #3721 is merged, I'm also enabling TSIM CI testing so we can make sure that TSIM support does not break moving forward. I've updated the title to reflect this major feature update.

[tmoreau89]

@vegaluisjose @liangfu please review, thanks!

[vegaluisjose]

Hey @tmoreau89 thanks for leading this.

Isn't going back to enable through CMAKE a similar thing than having it in the json file? this is how we used to have it initially. In my opinion is better to set any target in the same way and not have another level of indirection for a particular one. All targets sim, tsim, pynq, ultra96, f1, etc should all be built in the same way.

Regarding the simulation libraries (sim and tsim). I think it is better to actually build two shared libraries once somebody builds TVM like vta_sim.so and vta_tsim.so. We don't have to worry about any Verilator dependencies because we are using function pointers and Verilator dependency kicks in only when you want to build the shared library for chisel version.

Also, if we decided to go this route we could even have a way of generating a library for your target say vta_pynq.so or vta_your_fpga.so. The cool thing about this is that you don't have to rely on the config.json file for selecting target in python. Then, we could be able to select target directly from python without rebuilding again. Also, it is extremely straightforward to know what was the target of a particular library, which is not the case right now with a pure vta name.

[tmoreau89]

Thanks for the feedback Luis, I think we can build TSIM by default in either simulation modes then in order to always generate have both vta_sim.so and vta_tsim.so; and check upon runtime that the hardware library was built by the chisel make; and if not error out.

When in pynq, ultra96 etc. mode it's not necessary to build these simulation libraries, so I'll leave them out (since building the rpc_runtime already takes a while)

[tmoreau89]

@vegaluisjose I've made the change so that in either simulation modes, we always build both tsim and sim drivers. Then the hardware design that is integrated with the tsim driver will be determined by what is built inside of the chisel directory.

[liangfu]

@tmoreau89 This is a great idea with careful engineering work!

I would agree with @vegaluisjose to rename the libraries into libvta_sim.so and libvta_tsim.so. Even more aggressively, can we just provide the target options inside CMAKE, instead of parsing VTA_TARGET from the json file? Specifically, I think we can provide the options in the CMAKE like the following

option(build_vta_sim "Build VTA with fast simulation support." ON)
option(build_vta_tsim "Build VTA with cycle-accurate simulation support." ON)

, and add_library when the corresponding option has been enabled.

For the benefit of doing this, here I quote from @vegaluisjose 's comment,

we could be able to select target directly from python without rebuilding again.

[tmoreau89]

@liangfu this is an interesting suggestion; essentially control the build of runtime DLLs from the CMAKE, and control what hardware design gets emulated from the json file.

One suggestion would be to have a target string, like tsim, sim, or fpga since tsim has multiple targets that it can emulate. Thoughts @vegaluisjose @liangfu ?

[tmoreau89]

Actually, this may be a little more complicated due to the simple reason that depending on the target (say pynq vs. sim) we build the same DLL but with different driver sources. This lets us have a streamlined compilation both on host for simulation purposes, and on FPGA devices to build the RPC server / runtime.

[tmoreau89]

Therefore, if we want to have CMAKE controlled build for the simulator sources, we may need yet another parameter for the VTA RPC when we build on device (e.g. pynq). We'd essentially have 3 options:

# Whether to build fast VTA simulator driver
set(USE_VTA_FSIM ON)

# Whether to build cycle-accurate VTA simulator driver
set(USE_VTA_TSIM ON)

# Whether to build VTA RPC server (device side only)
set(USE_VTA_RPC OFF)

[liangfu]

One suggestion would be to have a target string, like tsim, sim, or fpga

This is fine by me.

since tsim has multiple targets that it can emulate.

I switched the testing between DefaultF1Config and DefaultPynqConfig without rebuilding libvta.so, because the 'multiple targets' are handled in libvta_hw.so with exactly the same interface. Therefore, there is no need to build multiple copies of libvta_tsim.so.

We'd essentially have 3 options: ...

I personally think this is a better design.

[tmoreau89]

OK sounds like we are on the same page! Please see the latest commit - this will let us explicitly set what dlls to build (either TSIM, FSIM, or FPGA driver); then the json target determines what FPGA is targeted. Right now it's only used when building the FPGA driver, but we could also use it to set the TSIM top-level design, e.g. DefaultF1Config or DefaultPynqConfig

[liangfu]

LGTM

[tmoreau89]

Note before merging, I'll need to update the docs to reflect the build steps for simulation or to run on FPGA. So please do not merge right away!

[tmoreau89]

@liangfu quick update: one suggestion that was made was to have a single parameter that indicates whether we are building on host or on FPGA: if it's on host, we build both simulator runtimes, and if it's on FPGA, we build the pynq runtime (or other if it's another vendor). How does that sound?

[tmoreau89]

I suggest we hold from merging this PR until #3721 is merged, and we can enable TSIM unit testing.

[liangfu]

@tmoreau89 I would personally prefer the explicit way for this, in order to configure what to build separately.

[tmoreau89]

Thanks for the feedback Liangfu.

[tmoreau89]

We'll stick with the 3 flags in the config.cmake

[tmoreau89]

#3721 is merged, and I've enabled TSIM CI testing!

[apivovarov]

@jroesch @tmoreau89

In file included from /root/tvm/vta/src/runtime.cc:29:0:
/root/tvm/vta/include/vta/hw_spec.h:104:33: error: 'VTA_LOG_UOP_BUFF_SIZE' was not declared in this scope
 #define VTA_LOG_UOP_BUFF_DEPTH (VTA_LOG_UOP_BUFF_SIZE - VTA_LOG_UOP_WIDTH + 3)

Can you explain where you declare in VTA_LOG_UOP_BUFF_SIZE
in https://github.com/dmlc/tvm/blob/master/vta/include/vta/hw_spec.h ?

[liangfu]

@apivovarov I think VTA_LOG_UOP_BUFF_SIZE is a variable to be customized, and it's defined in vta_config.json. Hope this helps.

[apivovarov]

@liangfu Can you reproduce the error? Try to build with LLVM

mkdir build
cd build
cp ../cmake/config.cmake .
cmake USE_LLVM=ON ..
make

Install TVM from Source https://docs.tvm.ai/install/from_source.html