Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

README.md: update Running User-Space Applications chapter #1358

Closed
JeanRochCoulon opened this issue Sep 4, 2023 · 25 comments
Closed

README.md: update Running User-Space Applications chapter #1358

JeanRochCoulon opened this issue Sep 4, 2023 · 25 comments
Assignees
Labels
Component:Doc For issues in the Documentation (e.g. for README.md files) Type:Enhancement For feature requests and enhancements

Comments

@JeanRochCoulon
Copy link
Contributor

JeanRochCoulon commented Sep 4, 2023

The Running User-Space Applications chapter need to be updated. As this chapter has been written before the GCC version update, the GCC variables are to be updated: riscv64-unknown-elf to be replaced by riscv-non-elf,...

The gate simulation chapter need also to be updated.

@JeanRochCoulon JeanRochCoulon added the Type:Bug For bugs in the RTL, Documentation, Verification environment or Tool and Build system label Sep 4, 2023
@zarubaf zarubaf added Type:Enhancement For feature requests and enhancements Component:Doc For issues in the Documentation (e.g. for README.md files) and removed Type:Bug For bugs in the RTL, Documentation, Verification environment or Tool and Build system labels Oct 23, 2023
@zarubaf zarubaf self-assigned this Oct 23, 2023
@rajatkmitra
Copy link

rajatkmitra commented Nov 4, 2023

What about the build ?? The most recent version of riscvpk attampt to run with riscv64-unknown-elf

@rajatkmitra
Copy link

Also for running user space applications this does not make any sense -
mkdir build
cd build
../configure --prefix=$RISCV --host=riscv64-unknown-elf
make
make install
if this is assuming the run directoy be cva6/
please expain where the build/ directory needs to be created

@rajatkmitra
Copy link

rajatkmitra commented Nov 5, 2023

The "Running User-Space Applications" section is completely misleading -

  1. It seems like the build/ directory needs to be -cva6/tmp/riscv-pk/build
  2. After running ../configure --prefix=$RISCV --host=riscv64-unknown-elf, the Makefile is created, and I coud make, make install
  3. Then make verilate fails as there is no verilate target in the Makefile
  4. Finally I am not sure what this means - work-ver/Variane_testharness $RISCV/riscv64-unknown-elf/bin/pk hello.elf

Please explain this section in as much details as possible.

@JeanRochCoulon
Copy link
Contributor Author

As you know open source is a collaboration database, made by inputs from all who would like to contribute. As you seem to know how to run pk, I would encourage you to submit a PR to explain better how to use it.

@JeanRochCoulon
Copy link
Contributor Author

Related to #1569

@rajatkmitra
Copy link

Yes, I reviewed #1569.. I am able to run the smoke and compliance tests. Everything worked nicely (I am using verilator but will try porting the design to Xcelium). My only concern was the notes under "Running User-Space Applications" was not very clear. Perhaps

  1. Please explain what directory you need to execute %make verilate in this section
  2. What is meant by %work-ver/Variane_testharness, is this a directory ?
    I am clear on the riscv-pk (proxy kernel) operation, just missing directlyies from where these need to be executed from

@JeanRochCoulon
Copy link
Contributor Author

@valentinThomazic

@valentinThomazic
Copy link
Contributor

@rajatkmitra verilate is a target of the Makefile at the root directory of the repo which means that it should be run from there.
work/ver/Variane_testharness is actually an emulator that will let you run your RISCV-V binary with the proxy kernel.

Please let me know if you manage to run your hello world (or not).

@rajatkmitra
Copy link

rajatkmitra commented Nov 7, 2023

Ah okay, thanks @valentinThomazic ..
So I try the following
% cd cva6 //this is the repo root
% which verilator
/usr/local/bin/verilator
% export CVA6_REPO_DIR=./
% pwd
% /home/rajatkmitra/risc5/pulp/cva6
%make verilate
Makefile:58: *** RISCV not set - please point your RISCV variable to your RISCV installation. Stop.

First RISCV is too generic a name, perhaps need to call this RISCVTOOLS to indicate the location of the toolchain...
Second there seems to not exist a directory called "work". Instead there is one called "work-ver"... And I did find the simulator inside it - Variane_testharness
Okay so then I searched to find when pk was installed, it was under -
/home/rajatkmitra/risc5/pulp/cva6/tmp/riscv-pk/build/pk
In here was also the hello.elf I compiled from prior steps

Then I ran like this -
rajatkmitra@butterfly:~/risc5/pulp/cva6$ work-ver/Variane_testharness /home/rajatkmitra/risc5/pulp/cva6/tmp/riscv-pk/build/pk hello.elf
But I did not get hello world ;-( , instead -

ajatkmitra@butterfly:~/risc5/pulp/cva6$ work-ver/Variane_testharness /home/rajatkmitra/risc5/pulp/cva6/tmp/riscv-pk/build/pk hello.elf
This emulator compiled with JTAG Remote Bitbang client. To enable, use +jtag_rbb_enable=1.
Listening on port 35223
*** [rvf_tracer] WARNING: No valid address of 'tohost' (tohost == 0x00000000000000), termination possible only by timeout or Ctrl-C!

rajatkmitra@butterfly:/risc5/pulp/cva6$ ^C
rajatkmitra@butterfly:
/risc5/pulp/cva6$ work-ver/Variane_testharness /home/rajatkmitra/risc5/pulp/cva6/tmp/riscv-pk/build/pk hello.elf +jtag_rbb_enable=1
This emulator compiled with JTAG Remote Bitbang client. To enable, use +jtag_rbb_enable=1.
Listening on port 43311
*** [rvf_tracer] WARNING: No valid address of 'tohost' (tohost == 0x00000000000000), termination possible only by timeout or Ctrl-C!
Attempting to accept client socket

It seems that the instructions need to be a little more specific.. Can you point me to how to set tohost ??
Thanks,
Raj

@valentinThomazic
Copy link
Contributor

Did you let the simulation finish (until timeout) ? What result did you get ?
You can set tohost by using +tohost_addr= at the end of your simulation launch command line (if you don't know what to enter, you should look the tohost address in pk with nm).

@rajatkmitra
Copy link

rajatkmitra commented Nov 8, 2023

I tried to find the tohost_addr in pk, but to_hostaddr symbol appears to be missing -
rajatkmitra@butterfly:~ /risc5/pulp/cva6$ nm ./tmp/riscv-pk/build/pk | grep tohost_addr
rajatkmitra@butterfly:~ /risc5/pulp/cva6$
Suggestions ??

@rajatkmitra
Copy link

rajatkmitra commented Nov 8, 2023

Oops the symbol is "tohost", trying your suggestion now, here is the output, I am not seeing the printf result on the screen -

rajatkmitra@butterfly:~ /risc5/pulp/cva6$ nm ./tmp/riscv-pk/build/pk | grep tohost
80003b2c t do_tohost_fromhost.constprop.0
8000c008 D tohost
rajatkmitra@butterfly:~/risc5/pulp/cva6$ work-ver/Variane_testharness ./tmp/riscv-pk/build/pk ./hello.elf +tohost_addr=8000c008
This emulator compiled with JTAG Remote Bitbang client. To enable, use +jtag_rbb_enable=1.
Listening on port 36749
[32640] %Warning: ariane_testharness.sv:676: Assertion failed in TOP.ariane_testharness.p_assert: B Response Errored
[32643] %Warning: ariane_testharness.sv:676: Assertion failed in TOP.ariane_testharness.p_assert: B Response Errored
[32646] %Warning: ariane_testharness.sv:676: Assertion failed in TOP.ariane_testharness.p_assert: B Response Errored
[32649] %Warning: ariane_testharness.sv:676: Assertion failed in TOP.ariane_testharness.p_assert: B Response Errored
...
The simulation did not end, I had to ^C out of the runtime
--Raj

@valentinThomazic
Copy link
Contributor

valentinThomazic commented Nov 9, 2023

If you had waited long enough, the result from the simulation would probably be a failure.

First of all, I would try using the pk located in your $RISCV/<host>/bin as it is installed there at the end of the build along with the bbl.

If it still does not work :

Did you build the RISCV toolchain following the readme or did you use a pre-built one ?
It seems that you used a pre-built one since you managed to build the pk using riscv64-unknown-elf as host.

I had the same warning messages and failed simulation because the toolchain provided by the get/build scripts in the repo compiles in 32 bits by default whereas Ariane is a RISCV64 simulator.
I would check the architecture with which the pk is built and your hello compiled even though you seem to have a riscv64 toolchain by default.

Try building the pk with :
../configure --prefix=$RISCV --host=riscv64-unknown-elf --with-arch=rv64imafdc_zicsr_zifencei

Try compiling your hello with :
$RISCV/bin/riscv64-unknown-elf-gcc -march=rv64imafdc_zicsr_zifencei -mabi=lp64d <path to hello.c> -o <dest path>

Then try starting a new simulation with your new pk and elf.

Please let me know if you make any progress or have new issues.

@rajatkmitra
Copy link

rajatkmitra commented Nov 10, 2023

Yes I was using a riscv compiler from another area.
Okay, I reinstalled cva6 top level ---

\git clone https://github.com/openhwgroup/cva6.git
\cd cva6
\git submodule update --init --recursive

I also created a local toolchain as per instructions -

Set environment variable RISCV to the desired installation location.
The toolchain can be installed in any user-writable directory.
export RISCV= /path/to/toolchain/installation/directory

Get the source code of toolchain components from public repositiories.
cd util/gcc-toolchain-builder
bash ./get-toolchain.sh

For the build prerequisites, see the local README.md.

\ Build and install the GCC toolchain.
\bash ./build-toolchain.sh $RISCV

The toolchain is built under $RISCV and is named with the following prefix -
riscv-none-elf *

Finally when I run these commands to create "pk"
rajatkmitra@butterfly:~ /risc5/pulp/cva6/ci$ ls
build check-tests.sh get-torture.sh install-fesvr.sh install-verilator.sh riscv-amo-tests.list riscv-fp-tests.list torture_make.patch
build-riscv-gcc.sh default.config gitlab-ci-emul.sh install-riscvpk.sh make-tmp.sh riscv-asm-tests.list riscv-mul-tests.list travis-ci-emul.sh
build-riscv-tests.sh float.config install-dtc.sh install-spike.sh path-setup.sh riscv-benchmarks.list setup.sh
rajatkmitra@butterfly:~ /risc5/pulp/cva6/ci$ cd ..
rajatkmitra@butterfly:~ /risc5/pulp/cva6$ ./ci/install-riscvpk.sh
./ci/install-riscvpk.sh: line 7: cd: /home/rajatkmitra/risc5/pulp/cva6/tmp: No such file or directory
rajatkmitra@butterfly:~/risc5/pulp/cva6$


Finally I noticed when you are asking to build "pk" -
mkdir build
cd build
../configure --prefix=$RISCV --host=riscv64-unknown-elf
make
make install
you are asking to configure using host=risc64-unknown-elf
But all the tools in the toolcahin are name with the prefix - riscv-none* and there host probably should have been called riscv-none-elf.. BTW, should not the host be the target architecture ? Not sure why it is pointing to a toolchain prefix.

@rajatkmitra
Copy link

rajatkmitra commented Nov 12, 2023

When I create manually create a cva6/tmp directory and then run -
rajatkmitra@butterfly:~ /risc5/pulp/cva6$ mkdir tmp/
rajatkmitra@butterfly:~ /risc5/pulp/cva6$ ./ci/install-riscvpk.sh

This eventually results in an error -
gcc -MMD -MP -Wall -Werror -D__NO_INLINE__ -mcmodel=medany -O2 -std=gnu99 -Wno-unused -Wno-attributes -fno-delete-null-pointer-checks -fno-PIE -DBBL_LOGO_FILE="bbl_logo_file" -DMEM_START=0x80000000 -fno-stack-protector -U_FORTIFY_SOURCE -DBBL_PAYLOAD="bbl_payload" -I. -I../pk -I../bbl -I../softfloat -I../dummy_payload -I../machine -I../util -c ../pk/file.c
gcc: error: unrecognized argument in option ‘-mcmodel=medany’
gcc: note: valid arguments to ‘-mcmodel=’ are: 32 kernel large medium small
Makefile:336: recipe for target 'file.o' failed
make: *** [file.o] Error 1

-mcmodel option does not support madany option unless you are expecting to use the risc5 toolchain gcc. In that case the tool should be using -
riscv-none-elf-gcc
that was installed as a part of the get and build process. However running install-riscvpk.sh, I noticed that the configure script is looking for the following -

hecking for gcc option to enable C11 features... none needed
checking for riscv64-unknown-elf-g++... no
checking for riscv64-unknown-elf-c++... no
checking for riscv64-unknown-elf-gpp... no
checking for riscv64-unknown-elf-aCC... no
checking for riscv64-unknown-elf-CC... no
checking for riscv64-unknown-elf-cxx... no
checking for riscv64-unknown-elf-cc++... no
checking for riscv64-unknown-elf-cl.exe... no
checking for riscv64-unknown-elf-FCC... no
checking for riscv64-unknown-elf-KCC... no
checking for riscv64-unknown-elf-RCC... no
checking for riscv64-unknown-elf-xlC_r... no
checking for riscv64-unknown-elf-xlC... no
checking for riscv64-unknown-elf-clang++... no
checking for g++... g++
checking whether the compiler supports GNU C++... yes
checking whether g++ accepts -g... yes

I suspect not finding the required gcc, the tool is defaulting back to system wide gcc which is not the proper compiler for cva6. Furthere as I mentioned previously, the toolchain should have created the compiler family with the -
riscv-none-elf-
prefix and used $RISCV to look for these.. Appears that there is come inconsistency with what the install-pk script expects and what the download and build script did
Get and Build looked like this -

Set environment variable RISCV to the desired installation location.

The toolchain can be installed in any user-writable directory.

export RISCV= /path/to/toolchain/installation/directory

Get the source code of toolchain components from public repositiories.

cd util/gcc-toolchain-builder
bash ./get-toolchain.sh

For the build prerequisites, see the local README.md.

Build and install the GCC toolchain.

bash ./build-toolchain.sh $RISCV

Return to the toplevel CVA6 directory.

cd -

I think the install-pk script needs to be fixed and made consistent with the get and build scipts. Can you fix this ?

@valentinThomazic
Copy link
Contributor

valentinThomazic commented Nov 13, 2023

In order to get the pk working, please follow the readme until the User-Space Application section and use the scripts to build your toolchain.
Replace the content of install-riscv-pk with :

#!/bin/sh

set -e
ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
PATH=$RISCV/bin:/bin:$PATH

if [ -d /$ROOT/tmp ]
then
    echo "tmp directory exists"
else
    echo "Creating $ROOT/tmp"
    mkdir $ROOT/tmp
fi

cd $ROOT/tmp

echo "Installing RISC-V Proxy Kernel and Boot Loader"
git clone https://github.com/riscv-software-src/riscv-pk.git
cd riscv-pk
mkdir -p build
cd build
../configure --prefix=$RISCV --host=riscv-none-elf --with-arch=rv64imafdc_zicsr_zifencei
make
make install

Follow this :

Running User-Space Applications

⚠️ Warning: Using the Makefile to run simulation is deprecated. See Github issue #1569.

It is possible to run user-space binaries on CVA6 with (RISC-V Proxy Kernel and Boot Loader) and (Ariane RISCV-V CPU).
If you built the toolchain using the scripts in util/gcc-toolchain-builder, RISC-V PK can be installed by running: ./ci/install-riscvpk.sh

Otherwise, you should follow the instruction on the RISCV-PK repo to build it for your toolchain.
If you do so, make sure you build a 64 bits version of the PK since the Ariane Simulator is a RISCV64 simulator.

If not already done, add verilator and the toolchain binaries into your PATH

export PATH=$PATH:$RISCV/bin:VERILATOR_INSTALL_DIR/bin

Then to run a RISC-V ELF using the Verilator model do:

echo '
#include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello CVA6!\\n");
    return 0;
}' > hello.c
# the binary to use depends on your toolchain
riscv-none-elf-gcc -march=rv64imafdc_zicsr_zifencei -mabi=lp64d hello.c -o hello.elf 
make verilate
work-ver/Variane_testharness $RISCV/riscv-none-elf/bin/pk hello.elf

@rajatkmitra
Copy link

Thanks, I will let you know shortly how things go..
--R

@valentinThomazic
Copy link
Contributor

Did you manage to make it work @rajatkmitra ?

@rajatkmitra
Copy link

rajatkmitra commented Nov 21, 2023

So sorry for the long absence, I was on another project .. I updated the install script install-riscv-pk as you mentioned, but now getting authentication error -
rajatkmitra@butterfly:~ /risc5/pulp/cva6$ ./ci/install-riscvpk.sh
./ci/install-riscvpk.sh: 4: ./ci/install-riscvpk.sh: Bad substitution
tmp directory exists
Installing RISC-V Proxy Kernel and Boot Loader
Cloning into 'riscv-pk'...
remote: Enumerating objects: 4204, done.
remote: Counting objects: 100% (261/261), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 4204 (delta 164), reused 242 (delta 159), pack-reused 3943
Receiving objects: 100% (4204/4204), 1.56 MiB | 11.57 MiB/s, done.
Resolving deltas: 100% (2999/2999), done.
--2023-11-21 20:11:30-- https://git.savannah.gnu.org/cgit/config.git/plain/config.sub
Resolving git.savannah.gnu.org (git.savannah.gnu.org)... 209.51.188.168, 2001:470:142::168
Connecting to git.savannah.gnu.org (git.savannah.gnu.org)|209.51.188.168|:443... connected.
ERROR: cannot verify git.savannah.gnu.org's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
Unable to locally verify the issuer's authority.
To connect to git.savannah.gnu.org insecurely, use `--no-check-certificate'.
rajatkmitra@butterfly:~/risc5/pulp/cva6$

@rajatkmitra
Copy link

rajatkmitra commented Nov 23, 2023

Okay I update the script -
wget https://git.savannah.gnu.org/cgit/config.git/plain/config.sub --no-check-certificate
wget https://git.savannah.gnu.org/cgit/config.git/plain/config.guess --no-check-certificate
and seems like the tools installed as required...

rajatkmitra@butterfly:~ /risc5/pulp/toolchain/riscv-none-elf/bin$ ll
total 14816
drwxrwxr-x 2 rajatkmitra rajatkmitra 4096 Nov 23 15:09 ./
drwxrwxr-x 5 rajatkmitra rajatkmitra 4096 Nov 9 19:41 ../
-rwxr-xr-x 2 rajatkmitra rajatkmitra 1173056 Nov 9 19:33 ar*
-rwxr-xr-x 2 rajatkmitra rajatkmitra 1756744 Nov 9 19:33 as*
-rwxr-xr-x 1 rajatkmitra rajatkmitra 59616 Nov 23 15:09 bbl*
-rwxr-xr-x 1 rajatkmitra rajatkmitra 5576 Nov 23 15:09 dummy_payload*
-rwxr-xr-x 4 rajatkmitra rajatkmitra 2003320 Nov 9 19:33 ld*
-rwxr-xr-x 4 rajatkmitra rajatkmitra 2003320 Nov 9 19:33 ld.bfd*
-rwxr-xr-x 2 rajatkmitra rajatkmitra 1159240 Nov 9 19:33 nm*
-rwxr-xr-x 2 rajatkmitra rajatkmitra 1299368 Nov 9 19:33 objcopy*
-rwxr-xr-x 2 rajatkmitra rajatkmitra 2022088 Nov 9 19:33 objdump*
-rwxr-xr-x 1 rajatkmitra rajatkmitra 73832 Nov 23 15:09 pk*
-rwxr-xr-x 2 rajatkmitra rajatkmitra 1173088 Nov 9 19:33 ranlib*
-rwxr-xr-x 2 rajatkmitra rajatkmitra 1098624 Nov 9 19:33 readelf*
-rwxr-xr-x 2 rajatkmitra rajatkmitra 1299368 Nov 9 19:33 strip*
rajatkmitra@butterfly:~/risc5/pulp/toolchain/riscv-none-elf/bin$

@rajatkmitra
Copy link

Then I do these commands -
%$RISCV/bin/riscv-none-elf-gcc -march=rv64imafdc_zicsr_zifencei -mabi=lp64d hello.c -o hello.elf

from cva6/ directory it is successful

Then
%make verilate (fails)

../corev_apu/tb/dpi/SimDTM.cc:4:10: fatal error: fesvr/dtm.h: No such file or directory
4 | #include <fesvr/dtm.h>
| ^~~~~~~~~~~~~
compilation terminated.
Variane_testharness.mk:70: recipe for target 'SimDTM.o' failed
make[1]: *** [SimDTM.o] Error 1
make[1]: *** Waiting for unfinished jobs....
../corev_apu/tb/ariane_tb.cpp:42:10: fatal error: fesvr/dtm.h: No such file or directory
42 | #include <fesvr/dtm.h>
| ^~~~~~~~~~~~~
compilation terminated.
Variane_testharness.mk:68: recipe for target 'ariane_tb.o' failed
make[1]: *** [ariane_tb.o] Error 1
rm Variane_testharness__ALL.verilator_deplist.tmp
make[1]: Leaving directory '/home/rajatkmitra/risc5/pulp/cva6/work-ver'
Makefile:582: recipe for target 'verilate' failed
make: *** [verilate] Error 2

@rajatkmitra
Copy link

@valentinThomazic - any thoughts on the verilator failure ??

@valentinThomazic
Copy link
Contributor

Hello,

As you may have noticed, the support for the proxy kernel in the CVA6 repo has been dropped.
You should know that the right (i.e. supported) way of simulating the CVA6 is by using the CVA6.py script which does not support running user-space applications.

That said, it looks like some headers files from spike cannot be found.
You might have an environment issue or Spike might not be installed.
Did you run the smoke-tests ?
By following the instructions here (that I slighlty updated just now) from scratch, you should be able to run the pk on the Verilator model.

@rajatkmitra
Copy link

Thanks for those updates! Yes, it was several weeks ago and all the smoke tests passed... I will recreate the environment with the new instructions and attempt to run.

@JeanRochCoulon
Copy link
Contributor Author

Pk is no more supported by this repository. The issue is close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component:Doc For issues in the Documentation (e.g. for README.md files) Type:Enhancement For feature requests and enhancements
Projects
None yet
Development

No branches or pull requests

4 participants