[#9797] DocDB: Avoid catalog version going backward during online upgrade

Summary:
Before and during a major YSQL version upgrade, through the heartbeat mechanism, the master provides current catalog versions to tservers from the PG11 catalog versions table. Note that DDLs are not allowed during a YSQL major version upgrade, and the PG15 catalogs are being updated by being set to semantic equivalents of PG11. To the PG11-only clients (tservers), there is only one valid catalog version state and that is the state when the upgrade began.

While going through the upgrade flow, the C++ upgrade tests keep the cluster running while the YB master is switched out of upgrade mode. When the master is switched out of upgrade mode in this way, it switches to providing the current catalog versions from the PG15 catalog versions table, which is fresh and sets everything as version 1. Thus tservers see the version number going down and probabilistically crash themselves with the following error message:

  Ignoring ysql db 13245 catalog version update: new version too old. New: 1, Old: 2, ignored count: 31

Fix this incorrect state by copying the contents of the PG11 catalog versions table to PG15, right after the PG15 catalog is established, and before returning success of the catalog upgrade.
Jira: DB-1591

Test Plan:
Jenkins

On MacOS arm64:
  ./yb_build.sh release --cxx-test pg15_upgrade-test

On AlmaLinux 8:
  ./yb_build fastdebug --gcc11 --sj
  pg15_tests/get_shell_test_specs.sh | grep upgrade | pg15_tests/run_tests.sh

Reviewers: hsunder

Reviewed By: hsunder

Subscribers: ybase, yql

Differential Revision: https://phorge.dev.yugabyte.com/D38961

Author: foucher

src/yb/master/ysql/ysql_initdb_major_upgrade_handler.cc

@@ -153,9 +153,25 @@ Status YsqlInitDBAndMajorUpgradeHandler::InitDBAndSnapshotSysCatalog(
 }
 
 void YsqlInitDBAndMajorUpgradeHandler::RunMajorVersionUpgrade(const LeaderEpoch& epoch) {
-  auto status = RunMajorVersionCatalogUpgrade(epoch);
-  WARN_NOT_OK(
-      MajorVersionCatalogUpgradeFinished(status, epoch), "Failed to run major version upgrade");
+  auto status = RunMajorVersionUpgradeImpl(epoch);
+  WARN_NOT_OK(MajorVersionCatalogUpgradeFinished(status, epoch),
+              "Failed to run major version upgrade");
+}
+
+Status YsqlInitDBAndMajorUpgradeHandler::RunMajorVersionUpgradeImpl(const LeaderEpoch& epoch) {
+  RETURN_NOT_OK(RunMajorVersionCatalogUpgrade(epoch));
+  RETURN_NOT_OK_PREPEND(UpdateCatalogVersions(epoch), "Failed to update catalog versions");
+  return Status::OK();
+}
+
+// pg_upgrade does not migrate the catalog version table, so we have to explicitly copy the contents
+// of the pre-existing catalog table to the PG15 version of the table.
+Status YsqlInitDBAndMajorUpgradeHandler::UpdateCatalogVersions(const LeaderEpoch& epoch) {
+  RETURN_NOT_OK(sys_catalog_.DeleteAllYsqlCatalogTableRows({kPgYbCatalogVersionTableId},
+                                                           epoch.leader_term));
+  RETURN_NOT_OK(sys_catalog_.CopyPgsqlTables({kPgYbCatalogVersionTableIdPg11},
+                                             {kPgYbCatalogVersionTableId}, epoch.leader_term));
+  return Status::OK();
 }
 
 Status YsqlInitDBAndMajorUpgradeHandler::RunMajorVersionCatalogUpgrade(const LeaderEpoch& epoch) {

src/yb/master/ysql/ysql_initdb_major_upgrade_handler.h

@@ -59,6 +59,10 @@ class YsqlInitDBAndMajorUpgradeHandler {
 
   void RunMajorVersionUpgrade(const LeaderEpoch& epoch);
 
+  Status RunMajorVersionUpgradeImpl(const LeaderEpoch& epoch);
+
+  Status UpdateCatalogVersions(const LeaderEpoch& epoch);
+
   Status RunMajorVersionCatalogUpgrade(const LeaderEpoch& epoch);
 
   // Runs the initdb process to create the initial ysql sys catalog and snapshot the sys_catalog if