bencher: add new tracker, torrust-tracker #191

josecelano · 2024-03-15T17:16:39Z

Relates to: #190 (comment)

Adds a new tracker to the bencher: https://github.com/torrust/torrust-tracker

The bencher runs load tests with 3 BitTorrent UDP tracker:

This PR adds the Torrust Tracker.

To run the bencher you need to:

Clone the aquatic repo and build some of the crates: aquatic_udp_load_test, aquatic_bencher and aquatic_udp.
Install the external trackers: opentracker, chihaya, torrust-tracker. They must be executable from any path.

How to Setup `opentracker`

You can make following the official documentation or sudo apt install opentracker for Ubuntu.

How to Setup `chihava`

Follow the documentation. You need to install Go (sudo apt install golang-go).

How to Setup `torrust-tracker`

Build:

git clone git@github.com:torrust/torrust-tracker.git
cd torrust-tracker
cargo build --release 
cp ./target/release/torrust-tracker ~/bin

Run:

TORRUST_TRACKER_PATH_CONFIG="./config.toml" torrust-tracker

Config file:

announce_interval = 120
db_driver = "Sqlite3"
db_path = "./sqlite3.db"
external_ip = "0.0.0.0"
inactive_peer_cleanup_interval = 600
log_level = "error"
max_peer_timeout = 900
min_announce_interval = 120
mode = "public"
on_reverse_proxy = false
persistent_torrent_completed_stat = false
remove_peerless_torrents = false
tracker_usage_statistics = false

[[udp_trackers]]
bind_address = "0.0.0.0:3000"
enabled = true

[[http_trackers]]
bind_address = "0.0.0.0:7070"
enabled = false
ssl_cert_path = ""
ssl_enabled = false
ssl_key_path = ""

[http_api]
bind_address = "127.0.0.1:1212"
enabled = false
ssl_cert_path = ""
ssl_enabled = false
ssl_key_path = ""

[http_api.access_tokens]
admin = "MyAccessToken"

[health_check_api]
bind_address = "127.0.0.1:1313"

How to Setup `aquatic`

sudo apt update && sudo apt upgrade -y
sudo apt-get install libhwloc-dev
cargo build --profile=release-debug --all-features -p aquatic_udp

NOTICE: libhwloc-dev is needed for io-uring UDP tracker feature

How to Setup and Run the Aquatic UDP load test

It's used by the Bencher.

Build:

cargo build --profile=release-debug -p aquatic_udp_load_test

Run:

./target/release-debug/aquatic_udp_load_test

How to Setup and Run the Bencher

Build:

cargo build --profile=release-debug -p aquatic_bencher

Run

./target/release-debug/aquatic_bencher udp

josecelano · 2024-03-15T17:23:36Z

Hi @greatest-ape I can run the aquatic UDP load test with the torrust-tracker:

TORRUST_TRACKER_PATH_CONFIG="./config.toml" torrust-tracker
./target/release-debug/aquatic_udp_load_test

but for some reason, it is not working with the bencher. But I have the same problem with other trackers. I get no results for some test cases:

- Average responses per second: 0
- Average tracker CPU utilization: 0%

For example:

$ ./target/release-debug/aquatic_bencher udp
# Benchmark report

Total number of load test runs: 72
Estimated duration: 0 hours, 44 minutes

## Tracker cores: 1 (cpus: 0,16)
### aquatic_udp run (socket workers: 1) (load test workers: 8, cpus: 8-15,24-31)
- Average responses per second: 363,320
- Average tracker CPU utilization: 95.6%
- Peak tracker RSS: 192.2 MiB
### aquatic_udp run (socket workers: 1) (load test workers: 12, cpus: 4-15,20-31)
- Average responses per second: 388,086
- Average tracker CPU utilization: 95.5%
- Peak tracker RSS: 192.2 MiB
### aquatic_udp (io_uring) run (socket workers: 1) (load test workers: 8, cpus: 8-15,24-31)
- Average responses per second: 392,732
- Average tracker CPU utilization: 95.5%
- Peak tracker RSS: 212.5 MiB
### aquatic_udp (io_uring) run (socket workers: 1) (load test workers: 12, cpus: 4-15,20-31)
- Average responses per second: 431,356
- Average tracker CPU utilization: 95.5%
- Peak tracker RSS: 215.6 MiB
### opentracker run (workers: 0) (load test workers: 8, cpus: 8-15,24-31)
- Average responses per second: 0
- Average tracker CPU utilization: 0%
- Peak tracker RSS: 1.9 MiB
### opentracker run (workers: 0) (load test workers: 12, cpus: 4-15,20-31)
- Average responses per second: 0
- Average tracker CPU utilization: 0%
- Peak tracker RSS: 1.9 MiB
### opentracker run (workers: 1) (load test workers: 8, cpus: 8-15,24-31)
- Average responses per second: 0
- Average tracker CPU utilization: 0%
- Peak tracker RSS: 1.9 MiB
### opentracker run (workers: 1) (load test workers: 12, cpus: 4-15,20-31)
- Average responses per second: 0
- Average tracker CPU utilization: 0%
- Peak tracker RSS: 1.9 MiB
### chihaya run () (load test workers: 8, cpus: 8-15,24-31)
- Average responses per second: 120,951
- Average tracker CPU utilization: 190%
- Peak tracker RSS: 898.7 MiB
### chihaya run () (load test workers: 12, cpus: 4-15,20-31)
- Average responses per second: 117,691
- Average tracker CPU utilization: 191%
- Peak tracker RSS: 878.1 MiB
### torrust-tracker run () (load test workers: 8, cpus: 8-15,24-31)
- Average responses per second: 0
- Average tracker CPU utilization: 0%
- Peak tracker RSS: 0 B
### torrust-tracker run () (load test workers: 12, cpus: 4-15,20-31)
- Average responses per second: 0
- Average tracker CPU utilization: 0%
- Peak tracker RSS: 0 B
## Tracker cores: 2 (cpus: 0-1,16-17)
### aquatic_udp run (socket workers: 2) (load test workers: 8, cpus: 8-15,24-31)

...

Maybe the torrust-tracker is not started correctly from the bencher. I'm trying to find out the problem.

josecelano · 2024-03-15T17:52:34Z

Hey @greatest-ape I found the problem. The order of the command arguments was wrong:

I changed this:

Ok(Command::new("taskset")
    .env("TORRUST_TRACKER_PATH_CONFIG", tmp_file.path())
    .arg("--cpu-list")
    .arg(vcpus.as_cpu_list())
    .arg(&command.torrust_tracker)
    .stdout(Stdio::piped())
    .stderr(Stdio::piped())
    .spawn()?)

to this:

Ok(Command::new("taskset")
    .arg("--cpu-list")
    .arg(vcpus.as_cpu_list())
    .env("TORRUST_TRACKER_PATH_CONFIG", tmp_file.path())
    .arg(&command.torrust_tracker)
    .stdout(Stdio::piped())
    .stderr(Stdio::piped())
    .spawn()?)

To be honest. I do not know why. But now I'm getting results for the torrust-tracker:

### torrust-tracker run () (load test workers: 8, cpus: 8-15,24-31)
- Average responses per second: 228,345
- Average tracker CPU utilization: 189%
- Peak tracker RSS: 193.3 MiB
### torrust-tracker run () (load test workers: 12, cpus: 4-15,20-31)
- Average responses per second: 223,238
- Average tracker CPU utilization: 189%
- Peak tracker RSS: 191.6 MiB

Regarding the opentracker maybe I'm also using an old version not compatible with the options you are using.

josecelano · 2024-03-19T18:25:03Z

Hi @greatest-ape I've written a new blog post in the Torrust site (https://torrust.com/) explaining how we use the Aquatic benchmarking tools with the Torrust Tracker (not published yet).

I have a question regarding the Bencher. I see it uses the UDP load test command, so I'm assuming that all announce requests use a different peer and a peer always uses the same infohash, is that right? If that's the case I would miss a test for many peers trying to announce the same torrent.

josecelano · 2024-03-19T21:26:46Z

I've tried to run a complete test but I'm getting some errors like:

panic: too many concurrent operations on a single file or socket (max 1048575)

I guess I have to increase some limits. The full output:

2024-03-19-bencher.txt

greatest-ape · 2024-03-20T06:07:41Z

Nice idea with a blog post!

No, it actually does actually test with more requests for some torrents.

Also, I added a review comment to your code.

Yeah, chihaya tends to crash under heavy load. There is an issue opened about it somewhere, probably in the chihaya or Go repos. In my recollection it is amazingly an actual Go runtime limitation that they didn’t want to fix.

Another thing of interest here is that the default CpuMode gives an unfair advantage to chihaya and torrust since they open one worker per thread, while the benchmark config for aquatic and opentracker opens one per core. You can see this by looking at the average CPU utilization stats for the lower core counts. This could be solved by adding entries to test with double the current worker count too for aquatic/opentracker (with medium priority so that they can be skipped when not needed to save time).

The reason why I haven’t yet is that the current setup is meant to enable somewhat fair testing on virtual machines where hyperthreads don’t really correspond to real hyperthreads, but for that to work, SubsequentOnePerPair mode must be used.

Adds a new tracker to the bencher. https://github.com/torrust/torrust-tracker

josecelano · 2024-03-20T09:18:08Z

Hi!

Nice idea with a blog post!

It's published now: https://torrust.com/benchmarking-the-torrust-bittorrent-tracker

No, it actually does actually test with more requests for some torrents.

OK.

Also, I added a review comment to your code.

Do you mean the comment on the issue? I don't see any comment in this PR.

Yeah, chihaya tends to crash under heavy load. There is an issue opened about it somewhere, probably in the chihaya or Go repos. In my recollection it is amazingly an actual Go runtime limitation that they didn’t want to fix.

Another thing of interest here is that the default CpuMode gives an unfair advantage to chihaya and torrust since they open one worker per thread, while the benchmark config for aquatic and opentracker opens one per core. You can see this by looking at the average CPU utilization stats for the lower core counts. This could be solved by adding entries to test with double the current worker count too for aquatic/opentracker (with medium priority so that they can be skipped when not needed to save time).

The reason why I haven’t yet is that the current setup is meant to enable somewhat fair testing on virtual machines where hyperthreads don’t really correspond to real hyperthreads, but for that to work, SubsequentOnePerPair mode must be used.

crates/bencher/src/protocols/udp.rs

greatest-ape · 2024-03-20T16:25:11Z

OK, do you see them now? Otherwise, I'm just wondering, does torrust-tracker set defaults for missing config keys? And in that case, can any be excluded here so that the bencher doesn't need to be updated if they are changed?

Also, could you please state that you provide the code that you're adding under the Apache 2.0 License? :-)

josecelano · 2024-03-21T14:33:23Z

OK, do you see them now? Otherwise, I'm just wondering, does torrust-tracker set defaults for missing config keys? And in that case, can any be excluded here so that the bencher doesn't need to be updated if they are changed?

I see them now. I've just repleid.

Also, could you please state that you provide the code that you're adding under the Apache 2.0 License? :-)

Yes, all the code in this PR is provided under the Apache 2.0 License.

greatest-ape · 2024-03-21T14:54:43Z

I don’t see your reply 😀

greatest-ape · 2024-03-21T15:32:13Z

OK, I checked, its seems like torrust-tracker does not have defaults for individual fields. I'm merging this.

greatest-ape · 2024-03-21T15:32:38Z

Also, could you please state that you provide the code that you're adding under the Apache 2.0 License? :-)

Yes, all the code in this PR is provided under the Apache 2.0 License.

Excellent.

josecelano · 2024-03-21T15:38:07Z

I don’t see your reply 😀

https://github.com/greatest-ape/aquatic/pull/191/files#r1527138893

greatest-ape · 2024-03-21T16:17:06Z

Alright :-)

Yeah, I want to add the ability to just run certain trackers at some point.

When you've refactored torrust-tracker configuration, could you please open a new PR for the bencher then? :-)

greatest-ape · 2024-03-21T16:18:30Z

I've merged some adjustments to make results more fair when running non-virtualized, so you might want to run your benchmarks again.

josecelano · 2024-03-21T16:33:20Z

Alright :-)

Yeah, I want to add the ability to just run certain trackers at some point.

When you've refactored torrust-tracker configuration, could you please open a new PR for the bencher then? :-)

Sure.

josecelano · 2024-03-21T16:37:19Z

I've merged some adjustments to make results more fair when running non-virtualized, so you might want to run your benchmarks again.

I will do it, and also again after finishing some improvements we are working on:

Small changes in the core tracker.
Replace the internal BTreeMap with a DashMap.

josecelano force-pushed the add-torrust-tracker-to-bencher branch from b8cdefe to dd1a5f9 Compare March 15, 2024 17:47

josecelano marked this pull request as ready for review March 15, 2024 17:52

josecelano mentioned this pull request Mar 15, 2024

Run benchmarking test with the aquatic bencher torrust/torrust-tracker#735

Closed

bencher: add new tracker, torrust-tracker

c5843ee

Adds a new tracker to the bencher. https://github.com/torrust/torrust-tracker

josecelano force-pushed the add-torrust-tracker-to-bencher branch from dd1a5f9 to c5843ee Compare March 20, 2024 09:11

greatest-ape requested changes Mar 20, 2024

View reviewed changes

crates/bencher/src/protocols/udp.rs Show resolved Hide resolved

greatest-ape approved these changes Mar 21, 2024

View reviewed changes

greatest-ape merged commit 8d51b6f into greatest-ape:master Mar 21, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bencher: add new tracker, torrust-tracker #191

bencher: add new tracker, torrust-tracker #191

josecelano commented Mar 15, 2024 •

edited

Loading

josecelano commented Mar 15, 2024 •

edited

Loading

josecelano commented Mar 15, 2024

josecelano commented Mar 19, 2024 •

edited

Loading

josecelano commented Mar 19, 2024

greatest-ape commented Mar 20, 2024 •

edited

Loading

josecelano commented Mar 20, 2024

greatest-ape commented Mar 20, 2024 •

edited

Loading

josecelano commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

josecelano commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

josecelano commented Mar 21, 2024

josecelano commented Mar 21, 2024

bencher: add new tracker, torrust-tracker #191

bencher: add new tracker, torrust-tracker #191

Conversation

josecelano commented Mar 15, 2024 • edited Loading

How to Setup opentracker

How to Setup chihava

How to Setup torrust-tracker

How to Setup aquatic

How to Setup and Run the Aquatic UDP load test

How to Setup and Run the Bencher

josecelano commented Mar 15, 2024 • edited Loading

josecelano commented Mar 15, 2024

josecelano commented Mar 19, 2024 • edited Loading

josecelano commented Mar 19, 2024

greatest-ape commented Mar 20, 2024 • edited Loading

josecelano commented Mar 20, 2024

greatest-ape commented Mar 20, 2024 • edited Loading

josecelano commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

josecelano commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

greatest-ape commented Mar 21, 2024

josecelano commented Mar 21, 2024

josecelano commented Mar 21, 2024

josecelano commented Mar 15, 2024 •

edited

Loading

How to Setup `opentracker`

How to Setup `chihava`

How to Setup `torrust-tracker`

How to Setup `aquatic`

josecelano commented Mar 15, 2024 •

edited

Loading

josecelano commented Mar 19, 2024 •

edited

Loading

greatest-ape commented Mar 20, 2024 •

edited

Loading

greatest-ape commented Mar 20, 2024 •

edited

Loading