Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated Test Suite Execution Documentation #24

Merged
merged 2 commits into from
Jul 4, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 61 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,10 +162,67 @@ $ ./configure --target=arc64 \
--enable-linux
```

### Running GCC Test Suite
## Running Test Suite

At present, the testing environment comprises three integrated testsuites: `gcc/g++`, `binutils`, and `newlib`. Testing is possible for both Baremetal Toolchain Distribution and Linux Toolchain Distribution (user-mode), via two simulators, QEMU and nSIM.

### Setting up the Simulator

During the toolchain compilation process, you can choose the simulator to use for testing. There are two options available: QEMU, an open-source simulator, and nSIM, a proprietary simulator provided by Synopsys. By default, QEMU is the chosen simulator.

1. If the selected simulator is QEMU, the testing will clone QEMU’s repository from the official Synopsys QEMU repository and build it by according to the branch specified in the Makefile file. The simulator will be installed based on the prefix path provided during the toolchain distribution’s configuration stage.
```sh
$ ./configure --target=... --prefix=/path/to/install --disable-linux
$ make newlib
$ make check-gcc
$ ./configure --target=... --prefix=/path/to/install --with-sim=qemu
```
2. If nSIM is preferred over QEMU, ensure that the simulator is defined in the PATH environment variable before executing the testing.
```sh
$ ./configure --target=... --prefix=/path/to/install --with-sim=nsim
```


### Reporting GCC/C++, Binutils, and Newlib

To execute toolchain tests, running the command `make report` will run all the tests in the GCC regression test suite. This command automatically invokes the GCC/G++, Binutils and Newlib testsuites to perform the respective tests. At the end of the execution, a report is generated by a testsuite filter script. This report filters known errors and displays only the unknown ones. Depending on the testsuite tool and CPU used, different filters with JSON format files are utilized. These files can be found in the `arc-gnu-toolchain/test/allowlist` folder.

To generate the report and perform the testing, use the following command:
```bash
$ make report -j$(nproc)
```
Running this command in parallel (e.g., “make report -j32”) will significantly speed up the execution time on multi-processor systems.

### Selecting the tests to run in GCC's regression test suite

By default, DejaGNU runs all tests in its regression test suite. However, executing all these tests can take a significant amount of time, which is impractical for typical development cycles. To address this, DejaGNU provides the option to select specific tests using the environment variable `RUNTESTFLAGS`.

For instance, if you want to run only the tests related to `tls`, you can use the following command:
```bash
RUNTESTFLAGS="tls.exp" make report
```

Likewise, if you wish to run a specific test, such as `thread_local-order2.C` within the `tls` tests, you can use the following command:
```bash
RUNTESTFLAGS="tls.exp=thread_local-order2.C" make report
```

These commands allow you to focus on specific test cases and streamline the execution time accordingly.



### Testing GCC/C++, Binutils, and Newlib

Alternatevely to the `report` command, in order to **only** execute all the associated test suite with the toolchain distribution, execute the following command:

```sh
$ make check -j $(nproc)
```

Refer to the table below to determine the suitable command for running an individual testsuite based on the current toolchain distribution.

| Toolchain Distribution | Test Suite | Command |
|:-----------------------|:-----------|:----------------------------------|
| Baremetal | gcc | `$ make check-gcc-baremetal` |
| Baremetal | binutils | `$ make check-binutils-baremetal` |
| Baremetal | newlib | `$ make check-newlib-baremetal` |
| Linux | gcc | `$ make check-gcc-linux` |
| Linux | binutils | `$ make check-binutils-linux` |

Loading