Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remote_boostrap_rate_limit_bytes_per_sec gflag name has a typo #1038

Closed
kmuthukk opened this issue Mar 20, 2019 · 0 comments
Closed

remote_boostrap_rate_limit_bytes_per_sec gflag name has a typo #1038

kmuthukk opened this issue Mar 20, 2019 · 0 comments
Assignees
@hectorgcr
Labels
kind/enhancement This is an enhancement of an existing feature

Comments

@kmuthukk
Copy link
Collaborator

We have this:

DEFINE_int64(remote_boostrap_rate_limit_bytes_per_sec, 100_MB,
             "Maximum transmission rate during a remote bootstrap. This is across all the remote "
             "bootstrap sessions for which this process is acting as a sender or receiver. So "
             "the total limit will be 2 * remote_boostrap_rate_limit_bytes_per_sec because a "
             "tserver or master can act both as a sender and receiver at the same time.");

remote_boostrap_rate_limit_bytes_per_sec
should be:
remote_bootstrap_rate_limit_bytes_per_sec

We can deprecate the old gflag, and introduce a new one that we document going forward.

We could change the old gflag's value to -1, and if some install explicitly overrides it, we can give that higher priority (just to not break compatibility).
@kmuthukk kmuthukk added the kind/enhancement This is an enhancement of an existing feature label Mar 20, 2019
yugabyte-ci pushed a commit that referenced this issue Mar 26, 2019
@hectorgcr
#1038: Change FLAGS_remote_boostrap_rate_limit_bytes_per_sec to FLAGS…
4a56fe1 
…_remote_bootstrap_rate_limit_bytes_per_sec

Summary:
Correcting a misspelling: boostrap -> bootstrap

The old flag has been deprecated to avoid crashes in case someone is still using it.

If the old flag is set, we use this value to set FLAGS_remote_bootstrap_rate_limit_bytes_per_sec and log a WARNING to indicate that the old flag has been deprecated.

Test Plan:
Created a new local cluster with:
```bin/yb-ctl --rf 3 create --tserver_flags "remote_boostrap_rate_limit_bytes_per_sec=1048576" --disable_ysql```

Started a workload on the yb-sample-apps repository to produce SST and WAL files that can be transferred during a remote bootstrap:
```java -jar target/yb-sample-apps.jar -workload CassandraKeyValue -num_threads_read 1 -num_threads_write 16 -nodes 127.0.0.1:9042,127.0.0.2:9042,127.0.0.3:9042 --num_reads 50000000 --num_unique_keys 1000000```

After a while, stopped the workload and added a new node:
```bin/yb-ctl add_node --tserver_flags "remote_boostrap_rate_limit_bytes_per_sec=1048576"```

Checked that the warning gets logged whenever tservers get started:
```W0325 12:59:30.240039 343795136 tablet_server_main.cc:136] Flag remote_boostrap_rate_limit_bytes_per_sec has been deprecated. Use remote_bootstrap_rate_limit_bytes_per_sec flag instead
W0325 13:03:15.105321 428012992 tablet_server_main.cc:136] Flag remote_boostrap_rate_limit_bytes_per_sec has been deprecated. Use remote_bootstrap_rate_limit_bytes_per_sec flag instead
W0325 12:59:30.337563 194676160 tablet_server_main.cc:136] Flag remote_boostrap_rate_limit_bytes_per_sec has been deprecated. Use remote_bootstrap_rate_limit_bytes_per_sec flag instead
W0325 12:59:30.282724 235877824 tablet_server_main.cc:136] Flag remote_boostrap_rate_limit_bytes_per_sec has been deprecated. Use remote_bootstrap_rate_limit_bytes_per_sec flag instead
```

Checked that the files transfers respected the rate set with the old flag:
```I0325 13:04:43.149715 296603648 remote_bootstrap_client.cc:657] Downloaded file MANIFEST-000009 of size 525 in 0.00169767 seconds
I0325 13:04:49.960532 296603648 remote_bootstrap_client.cc:657] Downloaded file 000008.sst.sblock.0 of size 4618157 in 6.81102 seconds
I0325 13:04:49.961621 296603648 remote_bootstrap_client.cc:657] Downloaded file CURRENT of size 16 in 0.00100847 seconds
I0325 13:04:50.170826 296603648 remote_bootstrap_client.cc:657] Downloaded file 000008.sst of size 204814 in 0.209176 seconds
I0325 13:04:51.217257 296603648 remote_bootstrap_client.cc:695] T b5ce99c34377439790c27637ae7bc092 P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 1 of size 1049597 in 1.0462 seconds
I0325 13:04:53.364852 296603648 remote_bootstrap_client.cc:695] T b5ce99c34377439790c27637ae7bc092 P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 2 of size 2097301 in 2.14521 seconds
I0325 13:04:57.614763 296603648 remote_bootstrap_client.cc:695] T b5ce99c34377439790c27637ae7bc092 P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 3 of size 4194349 in 4.24934 seconds
I0325 13:05:06.199493 296603648 remote_bootstrap_client.cc:695] T b5ce99c34377439790c27637ae7bc092 P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 4 of size 8390512 in 8.58342 seconds
I0325 13:05:23.776350 296603648 remote_bootstrap_client.cc:695] T b5ce99c34377439790c27637ae7bc092 P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 5 of size 16777475 in 17.5734 seconds
I0325 13:05:25.634744 296603648 remote_bootstrap_client.cc:695] T b5ce99c34377439790c27637ae7bc092 P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 6 of size 1859018 in 1.8392 seconds
I0325 13:05:27.154302 298749952 remote_bootstrap_client.cc:657] Downloaded file 000010.sst of size 67365 in 0.133192 seconds
I0325 13:05:27.156340 298749952 remote_bootstrap_client.cc:657] Downloaded file MANIFEST-000009 of size 1009 in 0.00199612 seconds
I0325 13:05:33.656746 298749952 remote_bootstrap_client.cc:657] Downloaded file 000008.sst.sblock.0 of size 4132708 in 6.50059 seconds
I0325 13:05:33.657721 298749952 remote_bootstrap_client.cc:657] Downloaded file CURRENT of size 16 in 0.000871587 seconds
I0325 13:05:33.861400 298749952 remote_bootstrap_client.cc:657] Downloaded file 000008.sst of size 204031 in 0.203643 seconds
I0325 13:05:34.405670 298749952 remote_bootstrap_client.cc:657] Downloaded file 000010.sst.sblock.0 of size 532112 in 0.544236 seconds
I0325 13:05:38.701637 298749952 remote_bootstrap_client.cc:695] T c8d68e02d8f24ca680eff097b5c8893c P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 3 of size 4194841 in 4.29573 seconds
I0325 13:05:47.254910 298749952 remote_bootstrap_client.cc:695] T c8d68e02d8f24ca680eff097b5c8893c P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 4 of size 8389160 in 8.54884 seconds
I0325 13:06:04.502209 298749952 remote_bootstrap_client.cc:695] T c8d68e02d8f24ca680eff097b5c8893c P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 5 of size 16777291 in 17.2435 seconds
I0325 13:06:06.448598 298749952 remote_bootstrap_client.cc:695] T c8d68e02d8f24ca680eff097b5c8893c P bbf677e3da8741d99eb4d31a5b380290: Remote bootstrap client: Downloaded WAL segment with seq. number 6 of size 1958976 in 1.94318 seconds
```

Add a new master with the old flag
```bin/yb-ctl add_node --master  --master_flags "remote_boostrap_rate_limit_bytes_per_sec=10240"```

Verify that the warning got logged in the masters
```W0325 12:59:30.109798 303494592 master_main.cc:102] Flag remote_boostrap_rate_limit_bytes_per_sec has been deprecated. Use remote_bootstrap_rate_limit_bytes_per_sec flag instead
W0325 13:10:58.896003 312034752 master_main.cc:102] Flag remote_boostrap_rate_limit_bytes_per_sec has been deprecated. Use remote_bootstrap_rate_limit_bytes_per_sec flag instead
W0325 12:59:30.184674 297227712 master_main.cc:102] Flag remote_boostrap_rate_limit_bytes_per_sec has been deprecated. Use remote_bootstrap_rate_limit_bytes_per_sec flag instead
W0325 12:59:30.148008 398136768 master_main.cc:102] Flag remote_boostrap_rate_limit_bytes_per_sec has been deprecated. Use remote_bootstrap_rate_limit_bytes_per_sec flag instead```

Verify that tablet 00000000000000000000000000000000 transfer respected the 10KB/s limit:

```I0325 13:10:59.150147 216600576 remote_bootstrap_client.cc:657] Downloaded file MANIFEST-000009 of size 562 in 0.0600203 seconds
I0325 13:11:00.240432 216600576 remote_bootstrap_client.cc:657] Downloaded file 000008.sst.sblock.0 of size 10759 in 1.09024 seconds
I0325 13:11:00.244314 216600576 remote_bootstrap_client.cc:657] Downloaded file CURRENT of size 16 in 0.0037769 seconds
I0325 13:11:06.966809 216600576 remote_bootstrap_client.cc:657] Downloaded file 000008.sst of size 66476 in 6.72265 seconds
I0325 13:11:16.101730 216600576 remote_bootstrap_client.cc:695] T 00000000000000000000000000000000 P 2450744601554564903f1e024dc54e5e: Remote bootstrap client: Downloaded WAL segment with seq. number 1 of size 90489 in 9.13435 seconds
```

Reviewers: mikhail, kannan, bogdan

Reviewed By: bogdan

Subscribers: ybase, bharat

Differential Revision: https://phabricator.dev.yugabyte.com/D6366
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hectorgcr hectorgcr

Labels
kind/enhancement This is an enhancement of an existing feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kmuthukk @hectorgcr