feat(dup): support shell set fail_mode and collector duplication ops

[neverchanje]

What problem does this PR solve?

• Support shell command set_dup_fail_mode. See the following shell example for details.
• Add perf-counters for QPS of duplication shipping and handling.

New perf-counters

• collector*app.pegasus*app.stat.duplicate_qps#<app_name>
• collector*app.pegasus*app.stat.dup_shipped_ops#<app_name>
• collector*app.pegasus*app.stat.dup_failed_shipping_ops#<app_name>

Check List

Tests

• Manual test (add detailed scripts or steps below)

>>> add_dup temp onebox2
adding duplication succeed [app: temp, remote: onebox2, appid: 2, dupid: 1587111546, freeze: false]

>>> set_dup_fail_mode temp 1587111546 skip
set duplication(1587111546) fail_mode (skip) for app temp succeed

>>> query_dup temp -d
duplications of app [temp] in detail:
{"1":{"create_ts":"2020-04-17 16:19:06","dupid":1587111546,"fail_mode":"FAIL_SKIP","remote":"onebox2","status":"DS_START"},"appid":2}

>>> set_dup_fail_mode temp 1587111546 slow
set duplication(1587111546) fail_mode (slow) for app temp succeed

>>> query_dup temp -d
duplications of app [temp] in detail:
{"1":{"create_ts":"2020-04-17 16:19:06","dupid":1587111546,"fail_mode":"FAIL_SLOW","remote":"onebox2","status":"DS_START"},"appid":2}

"onebox2" is the remote cluster to duplicate. To test whether the perf-counters work, we use pegasus-YCSB to load writes to the local onebox.

> ./start.sh load
YCSB Client 0.12.0-SNAPSHOT
Command line: -db com.yahoo.ycsb.db.PegasusClient -p pegasus.config=file://./pegasus.properties -s -P ./workload_pegasus -load
Loading workload...
Starting test.
...
2020-04-20 10:38:42:570 30 sec: 57019 operations; 1575.9 current ops/sec; est completion in 14 hours 36 minutes [INSERT: Count=15759, Max=1591295, Min=158, Avg=663.11, 90=488, 99=973, 99.9=100991, 99.99=371967] 

• Case 1: onebox2 doesn't set up.

• Case 2: onebox2 sets up.

On the receiver side (onebox2), the data duplicating it handles displayed like this:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation
• Need to be included in the release note

[hycdong]

I just can't understand your updates in this function, could you please explain to me?

[neverchanje]

Sure. I left a comment here. It was a bug. DUPLICATE can not participate in duplication like other normal writes. Because duplication is designed to be a directed edge flowing from master to slave. Forwarding will unintentionally add edges in this topology.

[acelyc111]

Rename to dup_qps? so the three metrics have the same pefix to identify.

[neverchanje]

It's not trivial because on replica side the perf-counter called replica*app.pegasus*duplicate_qps@<pid>. Therefore on the collector side, it must follow the same name.