[#2058] Add retry logic to pgbench

Summary:
       Patch Pulled from: https://www.postgresql-archive.org/WIP-Patch-Pgbench-Serialization-and-deadlock-errors-td5966663i40.html

       Client's run is aborted only in case of a serious error, for example, the
       connection with the backend was lost. Otherwise if the execution of SQL or meta
       command fails, the client's run continues normally until the end of the current
       script execution (it is assumed that one transaction script contains only one
       transaction).

       Transactions with serialization or deadlock failures are rolled back and
       repeated until they complete successfully or reach the maximum number of tries
       (specified by the --max-tries option) / the maximum time of tries (specified by
       the --latency-limit option). These options can be combined together; but if
       none of them are used, failed transactions are not retried at all. If the last
       transaction run fails, this transaction will be reported as failed, and the
       client variables will be set as they were before the first run of this
       transaction.

       If there're retries and/or errors their statistics are printed in the progress,
       in the transaction / aggregation logs and in the end with other results (all and
       for each script). A transaction error is reported here only if the last try of
       this transaction fails. Also retries and/or errors are printed per-command with
       average latencies if you use the appropriate benchmarking option
       (--report-per-command, -r) and the total number of retries and/or errors is not
       zero.

       If a failed transaction block does not terminate in the current script, the
       commands of the following scripts are processed as usual so you can get a lot of
       errors of type "in failed SQL transaction" (when the current SQL transaction is
       aborted and commands ignored until end of transaction block). In such cases you
       can use separate statistics of these errors in all reports.

       If you want to distinguish between failures or errors by type (including which
       limit for retries was violated and how far it was exceeded for the
       serialization/deadlock errors), use the pgbench debugging output created with
       the option --debug and with the debugging level "fails" or "all". The first
       variant is recommended for this purpose because with in the second case the
       debugging output can be very large.

Test Plan:
  yb-ctl create
  ./build/latest/postgres/bin/pgbench -i -s 3 postgres
  ./build/latest/postgres/bin/pgbench -P 5 -T 60 -n -N -r -c 16 -j 4 --max-tries 3 postgres

Reviewers: mihnea

Reviewed By: mihnea

Subscribers: mihnea, karthik, kannan, neha, rahuldesirazu, bogdan, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D7065

src/postgres/src/bin/pgbench/t/001_pgbench_with_server.pl

@@ -134,7 +134,8 @@ sub pgbench
 		qr{builtin: TPC-B},
 		qr{clients: 2\b},
 		qr{processed: 10/10},
-		qr{mode: simple}
+		qr{mode: simple},
+		qr{maximum number of tries: 1}
 	],
 	[qr{^$}],
 	'pgbench tpcb-like');
@@ -153,7 +154,7 @@ sub pgbench
 	'pgbench simple update');
 
 pgbench(
-	'-t 100 -c 7 -M prepared -b se --debug',
+	'-t 100 -c 7 -M prepared -b se --debug all',
 	0,
 	[
 		qr{builtin: select only},
@@ -546,6 +547,11 @@ sub pgbench
 		q{-- MAX_ARGS=10 for prepared
 \set i 0
 SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
+}
+	],
+	[   'sql division by zero', 0, [qr{ERROR:  division by zero}],
+		q{-- SQL division by zero
+SELECT 1 / 0;
 }
 	],
 
@@ -720,6 +726,17 @@ sub pgbench
 		[qr{unrecognized time unit}], q{\sleep 1 week}
 	],
 
+	# CONDITIONAL BLOCKS
+	[   'if elif failed conditions', 0,
+		[qr{division by zero}],
+		q{-- failed conditions
+\if 1 / 0
+\elif 1 / 0
+\else
+\endif
+}
+	],
+
 	# MISC
 	[
 		'misc invalid backslash command',         1,
@@ -738,14 +755,33 @@ sub pgbench
 	my $n = '001_pgbench_error_' . $name;
 	$n =~ s/ /_/g;
 	pgbench(
-		'-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+		'-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared -d fails',
 		$status,
-		[ $status ? qr{^$} : qr{processed: 0/1} ],
+		($status ?
+		 [ qr{^$} ] :
+		 [ qr{processed: 0/1}, qr{number of errors: 1 \(100.000%\)},
+		   qr{^((?!number of retried)(.|\n))*$} ]),
 		$re,
 		'pgbench script error: ' . $name,
 		{ $n => $script });
 }
 
+# reset client variables in case of failure
+pgbench(
+	'-n -t 2 -d fails', 0,
+	[ qr{processed: 0/2}, qr{number of errors: 2 \(100.000%\)},
+	  qr{^((?!number of retried)(.|\n))*$} ],
+	[ qr{(client 0 got a failure in command 1 \(SQL\) of script 0; ERROR:  syntax error at or near ":"(.|\n)*){2}} ],
+	'pgbench reset client variables in case of failure',
+	{	'001_pgbench_reset_client_variables' => q{
+BEGIN;
+-- select an unassigned variable
+SELECT :unassigned_var;
+\set unassigned_var 1
+END;
+}
+	});
+
 # zipfian cache array overflow
 pgbench(
 	'-t 1', 0,

src/postgres/src/bin/pgbench/t/002_pgbench_no_server.pl

@@ -61,7 +61,7 @@ sub pgbench_scripts
 	# name, options, stderr checks
 	[
 		'bad option',
-		'-h home -p 5432 -U calvin -d --bad-option',
+		'-h home -p 5432 -U calvin -d all --bad-option',
 		[qr{--help.*more information}]
 	],
 	[
@@ -153,6 +153,11 @@ sub pgbench_scripts
 			qr{error while setting random seed from --random-seed option}
 		]
 	],
+	[
+		'bad maximum number of tries',
+		'--max-tries -10',
+		[ qr{invalid number of maximum tries: "-10"} ]
+	],
 
 	# loging sub-options
 	[