[YSQL] Allow running YSQL upgrade unit tests with snapshot other than 2.0.9.0 #23033
Closed
1 task done
Assignees
Labels
area/ysql
Yugabyte SQL (YSQL)
kind/enhancement
This is an enhancement of an existing feature
priority/medium
Medium priority issue
Comments
yugabyte-ci
added
kind/enhancement
myang2021
added a commit
that referenced
this issue
Jul 1, 2024
…an 2.0.9.0 Summary: Currently, when running some YSQL upgrade unit tests (e.g., org.yb.pgsql.TestYsqlUpgrade#migratingIsEquivalentToReinitdb), we create the cluster using initial_sys_catalog_snapshot_2.0.9.0_release or initial_sys_catalog_snapshot_2.0.9.0_debug. When doing investigation of 128-byte PG name (by changing NAMEDATALEN from 64 to 128, some YSQL upgrade unit tests failed (including `migratingIsEquivalentToReinitdb`). This is because the snapshot for 2.0.9.0 isn't compatible with the new 128-byte build. In order to run the test we need a compatible 2.0.9.0 snapshot that is also built with 128-byte NAMEDATALEN. But our build tools have changed and a simple git reset back to 2.0.9.0 to do a 128-byte NAMEDATALEN build did not work. However I was able to make a 2.14 128-byte NAMEDATALEN build which produced a compatible 2.14 snapshot. But in order to run YSQL upgrade test using 2.14 snapshot instead of 2.0.9.0 snapshot, I had to change the test code. It will be good to provide a java test argument that can be used to override the default 2.0.9.0 snapshot when running the YSQL upgrade test. I made a change to allow using --java-test-args to pass the absolute path to the snapshot directory to override the 2.0.9.0 snapshot. I found several tests had issue so made some adjustments needed to get them to pass. In particular: 1. Created a helper function `createPgTablegroupTableIfNotExists`: needed for both `migratingIsEquivalentToReinitdb` and `upgradeCheckingIdempotency`. 2. Some of the number calculations assumed 2.0.9.0 settings and are hard coded, they need to be adjusted when a different snapshot is used. Test Plan: (1) The default test PASS ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade' Note that it uses 2.0.9.0 snapshot to start upgrade (2) The customized test that uses 2.14.13.0 snapshot to start upgrade Download yugabyte-2.14.13.0-b13-linux-x86_64.tar to unpack and get a 2.14.13.0 snapshot at /tmp/yugabyte-2.14.13.0/share/initial_sys_catalog_snapshot. PASS ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.14.13.0/share/initial_sys_catalog_snapshot' (3) Repeated the test steps by making two 128-byte NAMEDATALEN builds with a change like: -#define NAMEDATALEN 64 +#define NAMEDATALEN 128 One for master branch, one for 2.14. The latter is used to produce a 128-byte NAMEDATALEN 2.14 snapshot that is compatible with the 128-byte NAMEDATALEN master build to test YSQL upgrade. (3.1) The default test fails as expected because the default snapshot 2.0.9.0 was built with 64-byte NAMEDATALEN and is not compatible with a 128-byte build. FAIL ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade' (3.2) Run the customized test using the 128-byte 2.14 snapshot. PASS ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/ysql-sys-catalog-snapshots-128/initial_sys_catalog_snapshot_2.14_release' Reviewers: yguan Reviewed By: yguan Subscribers: yql Differential Revision: https://phorge.dev.yugabyte.com/D36198
jasonyb
pushed a commit
that referenced
this issue
Jul 2, 2024
Summary: f8e73e9 [#18192] YSQL: Introduce interface to mock tserver response in MiniCluster tests 4ae68f4 Build break fix for centos7 Excluded: 2ec9224 [#23033] Allow running YSQL upgrade unit tests with snapshot other than 2.0.9.0 37912f1 [#22058] docdb: Disable connections on cloned db until cloning is complete 059b855 [#22908] xCluster: Use XClusterRemoteClient across XCluster 5dc5ee7 [#22849] YSQL: Correctly handle reset phase timeout errors in YSQL Connection Manager af49a1e [#22876][#22835][#22773] CDCSDK: Add new auto flag to identify non-eligible tables in CDC stream f3c4e14 [PLAT-14524] Up-version pekko to fix TLSActor infinite loop 9388aea [#23052] yugabyted: Restarting a node fails when data_dir is missing in user specified configuration. 5cf9736 [PLAT-12685]: Generate a YBA metric for xcluster config table status. 73fc90a [PLAT-14497]: Fix incremental backup time when none full backup exists e9b5ba5 [PLAT-14533]: Modify the gflags metadata support db version check 8dca952 [PLAT-14432][Platform] Show certificate Database Node Certificate/key and Client Certificate/key for CA certs in certificate details modal 6551e45 Add utkarsh.munjal to contributors.md bafa1cb [#21751] YSQL, ASH: Sampling of wait events Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, tfoucher Subscribers: yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36325
myang2021
added a commit
that referenced
this issue
Jul 2, 2024
…ests with snapshot other than 2.0.9.0 Summary: Original commit: 2ec9224 / D36198 This involves conflict with YB pg15 4638d7d on TestYsqlUpgrade.java migratingIsEquivalentToReinitdb @ignore. Currently, when running some YSQL upgrade unit tests (e.g., org.yb.pgsql.TestYsqlUpgrade#migratingIsEquivalentToReinitdb), we create the cluster using initial_sys_catalog_snapshot_2.0.9.0_release or initial_sys_catalog_snapshot_2.0.9.0_debug. When doing investigation of 128-byte PG name (by changing NAMEDATALEN from 64 to 128, some YSQL upgrade unit tests failed (including `migratingIsEquivalentToReinitdb`). This is because the snapshot for 2.0.9.0 isn't compatible with the new 128-byte build. In order to run the test we need a compatible 2.0.9.0 snapshot that is also built with 128-byte NAMEDATALEN. But our build tools have changed and a simple git reset back to 2.0.9.0 to do a 128-byte NAMEDATALEN build did not work. However I was able to make a 2.14 128-byte NAMEDATALEN build which produced a compatible 2.14 snapshot. But in order to run YSQL upgrade test using 2.14 snapshot instead of 2.0.9.0 snapshot, I had to change the test code. It will be good to provide a java test argument that can be used to override the default 2.0.9.0 snapshot when running the YSQL upgrade test. I made a change to allow using --java-test-args to pass the absolute path to the snapshot directory to override the 2.0.9.0 snapshot. I found several tests had issue so made some adjustments needed to get them to pass. In particular: 1. Created a helper function `createPgTablegroupTableIfNotExists`: needed for both `migratingIsEquivalentToReinitdb` and `upgradeCheckingIdempotency`. 2. Some of the number calculations assumed 2.0.9.0 settings and are hard coded, they need to be adjusted when a different snapshot is used. Test Plan: (1) The default test PASS ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade' Note that it uses 2.0.9.0 snapshot to start upgrade (2) The customized test that uses 2.14.13.0 snapshot to start upgrade Download yugabyte-2.14.13.0-b13-linux-x86_64.tar to unpack and get a 2.14.13.0 snapshot at /tmp/yugabyte-2.14.13.0/share/initial_sys_catalog_snapshot. PASS ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.14.13.0/share/initial_sys_catalog_snapshot' (3) Repeated the test steps by making two 128-byte NAMEDATALEN builds with a change like: -#define NAMEDATALEN 64 +#define NAMEDATALEN 128 One for master branch, one for 2.14. The latter is used to produce a 128-byte NAMEDATALEN 2.14 snapshot that is compatible with the 128-byte NAMEDATALEN master build to test YSQL upgrade. (3.1) The default test fails as expected because the default snapshot 2.0.9.0 was built with 64-byte NAMEDATALEN and is not compatible with a 128-byte build. FAIL ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade' (3.2) Run the customized test using the 128-byte 2.14 snapshot. PASS ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/ysql-sys-catalog-snapshots-128/initial_sys_catalog_snapshot_2.14_release' Reviewers: jason, tfoucher Reviewed By: jason Subscribers: yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36335
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-11961
Description
Currently, when running some YSQL upgrade unit tests (e.g.,
org.yb.pgsql.TestYsqlUpgrade#migratingIsEquivalentToReinitdb), we create the
cluster using initial_sys_catalog_snapshot_2.0.9.0_release or
initial_sys_catalog_snapshot_2.0.9.0_debug. When doing a test with changing
NAMEDATALEN from 64 to 128, some YSQL upgrade unit tests failed (including
migratingIsEquivalentToReinitdb). This is because the snapshot for 2.0.9.0 isn't
compatible with the new 128-byte build. In order to run the test we need a
compatible 2.0.9.0 snapshot that is also built with 128-byte NAMEDATALEN.
But our build tools have changed and a simple git reset back to 2.0.9.0 to do a
128-byte NAMEDATALEN build did not work.
However I was able to make a 2.14 128-byte NAMEDATALEN build which produced a
compatible 2.14 snapshot. But in order to run YSQL upgrade test using 2.14
snapshot instead of 2.0.9.0 snapshot, I had to change the test code. It will be
good to provide a java test argument that can be used to override the default
2.0.9.0 snapshot when running the YSQL upgrade test.
Issue Type
kind/enhancement
Warning: Please confirm that this issue does not contain any sensitive information
The text was updated successfully, but these errors were encountered: