dev/release#16 - Allow omission of empty upgrade steps

[totten]

Overview

This PR prepares to streamline the upgrader steps - by making it possible to skip increments which do not actually require DB updates.

(A companion PR will be opened momentarily to remove a large batch of old/unnecessary increments.)

ping @colemanw re: https://lab.civicrm.org/dev/release/-/issues/16

Before

(General) Often, when setting up the initial X.Y.beta1 and when setting up the final X.Y.0 stable release, there are no upgrade steps. However, we're required to create the files anyway to ensure that the upgrader continues to work normally. As a byproduct, we create a large number of empty incremental steps. Each incremental step requries 3x AJAX calls in the web-based upgrader -- so this adds up.

(Technical) The incremental revision sequence -- and the final DB version-number -- are strictly determined by the list of X.Y.Z.mysql.tpl files. If you omit a SQL file, then it will not execute corresponding PHP code, and it will not set
the final version in DB appropriately.

After

(General) The upgrader does not need SQL files for every X.Y.Z revision.

(Technical) The incremental revision sequence is determined by scanning both the list of X.Y.Z.mysql.tpl files and the list of upgrade_X_Y_Z() functions. It is valid to have a version which uses both, either, or neither. In cases where neither style of increment is used, it will still set the final version appropriately.

Technical Details

Writing upgrade steps has always been a bit particular. This is now slightly more robust, but some quirks remain, so it may be worth summarizing the resulting behavior. You may do any of these::

• (As before) Define vX.Y.Z with only a file X.Y.Z.mysql.tpl. The SQL script is called automatically.
• (New) Define vX.Y.Z with only a upgrade_X_Y_Z() step. The PHP function is called automatically.
• (As before) Define vX.Y.Z with both function upgrade_X_Y_Z() and file X.Y.Z.mysql.tpl. The flow-control for PHP takes precedence, and it decides if/when to call the SQL file.
• (New) Define vX.Y.Z without any incremental upgrade steps. This only makes sense for the HEAD version. The UI+DB will be updated to reflect the new number, but no incremental steps are actually executed.

This allows us to change the default policy (set-version.php) with respect to initializing upgrade files:

• Whenever we start the X.Y.alpha1 release, it will create a skeletal FiveUmpteen.php, create a skeletal X.Y.alpha1.mysql.tpl, and bump xmll/version.xml. (Most schema changes occur during alpha1.)
• Whenever we start the X.Y.beta1 or X.Y.0 release, it will only bump up xml/version.xml. It does not (by default) create any PHP or SQL files.

Comments

This slightly affects the workflow for a developer who seeks to change the upgrader during RC/beta/stable. (They need to manually bump the version and create any SQL files -- there is no pre-existing skeleton.) When this is merged, we need to send a PSA to civicrm-dev@.

If you want to inspect what it does with different commits (eg pre/post patch) or with some local hacks to the upgrade-steps, then these commands may be interesting:

## Print the list of incremental revisions
cv ev 'return (new CRM_Upgrade_Form())->getRevisionSequence();'

## Preview the upgrade tasks
cv upgrade:db -vv --dry-run

## Run the upgrade tasks
cv upgrade:db -vv

## Run the upgrade test using DB snapshots from 4.6.36
civibuild upgrade-test dmaster 4.6.36*

I have not done much in the way of end-to-end validation. There should be a question-mark with respect to 4.4.7 -- that file does not load right now (because the loading process requires the currently-non-existent FourSeven.php).

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[eileenmcnaughton]

I did the following test

• downloaded 4.5 tarball from sourcecode
• imported civicrm.mysql & civicrm_generated.mysql
• ran the upgrade
• exported the mysql file
• did the above steps with this patch applied
• compared the files

the only differences are dates in the tables

• civicrm_activity
• civicrm_log
• civicrm_settings
• civicrm_cache

Based on the above this 'does no harm' and I'm going to merge it. However, the above test did not pass on the fail on patch (there was a hard fail) so there might be another step of so in there