Upgrade Cumulus from v18.2.0 to v18.5.x

[krisstanton]

// Required - Completion or near completion of ticket #384

• Perform Cumulus Upgrade from v18.2.0 to v18.5.2 (or 18.5.3)
  • Ref ticket Research Cumulus Upgrade and make stable plan here #384
  • Upgrade from 18.2.0 to 18.3.3 (Requires update to NodeJS 20+)
    • Update Versions (Cumulus, NodeJS)
    • Upgrade Sandbox ---> Smoke Test
    • Upgrade UAT ---> Smoke Test
    • Upgrade PROD (after UAT Smoke Test Passes)
  • Upgrade from 18.3.3 to 18.4.0
    • Complete Ticket: Upgrade Cumulus from v18.3.3 to v18.4.0 #434
  • Upgrade from 18.4.0 to 18.5.2 (and possibly 18.5.3)
    • Complete Ticket: Upgrade Cumulus from v18.4.0 to v18.5.2 (If possible, 18.5.3) #435
  • Proceed to ticket Upgrade ORCA from v9.0.5 to v10.1.0 #431 to do the Upgrade for ORCA

[krisstanton]

WIP UPDATE:
Due to the nature of all the dependency and Database updates between the cumulus versions, I decided to go with a step-wise upgrade process. (18.2.0 --> 18.3.3 --> 18.4.0 --> 18.5.0)

• Attempted a Sandbox Deploy (a few times)
  • doing additional code changes between each attempt and documenting.
  • Currently tracking down an aws-sdk update as it relates to some of our type script stuff

terraspace up cumulus:  Error: failed to execute "/bin/bash": $ run-s build test lambda:install
terraspace up cumulus:  Error: Cannot find module 'aws-sdk'

[krisstanton]

WIP Update:
I am currently running into a small problem between the way Cumulus DB connections are handled for v18.3.3 and ORCA v9.0.5.
I am working through this problem at this time and may have a solution. Still getting the syntax right for my test.
If you are reading this, I have omitted the detailed description due to this being public and the upgrade being in progress.
If you need more detail on the specific nature of the problem while I'm still working on the upgrade, please let me know. If not, I plan on coming back and describing the approach, and the fix, after it has been implemented.

[krisstanton]

WIP Update:
After a bit of back and forth, I was able to get around the ORCA Migration problem that occurs during the upgrade from Cumulus v18.2.0 to v18.3.3 and Orca v9.0.5.

Now there is an issue with deploying to UAT. Note: This is AFTER a successful deploy plus smoke test in Sandbox and while trying to do the automated deployment to UAT via github actions (at the step plan-uat)

Run yarn --non-interactive install
node:internal/modules/cjs/loader:1146
  throw err;
  ^

Error: Cannot find module '/home/runner/work/csdap-cumulus/csdap-cumulus/.yarn/releases/yarn-1.22.22.cjs'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1143:15)
    at Module._load (node:internal/modules/cjs/loader:984:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:[12](https://github.com/NASA-IMPACT/csdap-cumulus/actions/runs/12586431153/job/35080230561?pr=432#step:12:13))
    at node:internal/main/run_main_module:28:49 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v20.12.2
Error: Process completed with exit code 1.

[krisstanton]

WIP Update:
This applies to the step from 18.2.0 to 18.3.3
Adding a bit of context to the above errors

• For the yarn error above
• • The solution was to also include the compiled file yarn-1.22.22.cjs with the deployment.
• There was an Error that looked to be related to aws/sdk, which only presented on local machines trying to do sandbox deployments.
• • The reality of this was that something was going wrong with docker during the upgrade of NodeJS from version 16 to 20+ To fix that error locally, here are the instructions.

// On a local machine
// (1) Exit any Cumulus related Docker Process
// (2) Go to your Docker UI and stop any containers that are related to Cumulus
// (3) Go to the Docker Volumes area and delete any volumes related to Cumulus
// (4) Go back to Terminal and run the following command, which will recreate the volumes and docker image including installing the correct versions of NodeJS without any conflicts
DOTENV=.env.sandbox make bash
// (5) Build and Run All tests which should now pass with this command
yarn test

• For the db connection problem with ORCA v9.0.5 and Cumulus 18.3.3, the solution was syntax related and can be found in the pull requests.

Current Upgrade Progress: Working on manual Database upgrades for the upgrade step from v18.3.3 to v18.4.0

[krisstanton]

WIP Update:
For the Step of 18.3.3 to 18.4.0
Currently in progress on the second of 2 database upgrades prior to deployment.

// Background
There are 2 long running database queries that have to be run prior to successful deployment.
I've managed to get one of them to run via local terminal.
The second process involves a query which is estimated to take more than a full 24 hours to run.
Everything is secured behind network interfaces and private subnets making this access a bit complex.

// Current
I am still in the process of getting a connection to the Database from inside that VPC.

// Next
Once the above is done and working in sandbox, I plan on adding this to the CSDA Cumulus Documentation (which will go into the code repo on the next pull request) so that next time around there will be smoother instructions to follow.