Upgrade dependencies 2024-06-24 (#6356, #6289, #6354)

[dsotirho-ucsc]

Connected issue: #6356, #6289, #6354

Checklist

Author

• Target branch is develop
• Name of PR branch matches upgrades/yyyy-mm-dd
• On ZenHub, PR is connected to the upgrade issue it resolves
• PR title matches Upgrade dependencies yyyy-mm-dd
• PR title references the connected issue

Author (upgrading deployments)

• Ran make image_manifests.json and committed the resulting changes _{or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable}
• Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading deployments}
• Added u tag to commit title _{or this PR does not require upgrading deployments}
• This PR is labeled upgrade _{or does not require upgrading deployments}
• This PR is labeled deploy:shared _{or does not modify image_manifests.json, and does not require deploying the shared component for any other reason}
• This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
• This PR is labeled backup:gitlab
• This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (before every review)

• Rebased PR branch on develop, squashed old fixups
• Ran make requirements_update _{or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile}
• Added R tag to commit title _{or this PR does not modify requirements*.txt}
• This PR is labeled reqs _{or does not modify requirements*.txt}
• make integration_test passes in personal deployment _{or this PR does not modify functionality that could affect the IT outcome}

System administrator (after approval)

• Actually approved the PR
• Labeled connected issue as no demo
• A comment to this PR details the completed security design review
• PR title is appropriate as title of merge commit
• Moved ticket to Approved column
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Squashed PR branch and rebased onto develop
• Sanity-checked history
• Pushed PR branch to GitHub
• Manually deleted the cerebro, kibana, and gitlab-runner images from dev.
• Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Made a backup of the GitLab data volume in dev (see operator manual for details) _{or this PR is not labeled backup:gitlab}
• Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Manually deleted the cerebro, kibana, and gitlab-runner images from anvildev.
• Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Made a backup of the GitLab data volume in anvildev (see operator manual for details) _{or this PR is not labeled backup:gitlab}
• Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Checked the items in the next section _{or this PR is labeled deploy:gitlab}
• PR is assigned to only the system administrator _{or this PR is not labeled deploy:gitlab}

System administrator

• Background migrations for dev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• Background migrations for anvildev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Ran _select dev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Ran _select anvildev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Added sandbox label
• Pushed PR branch to GitLab dev
• Pushed PR branch to GitLab anvildev
• Build passes in sandbox deployment
• Build passes in anvilbox deployment
• Reviewed build logs for anomalies in sandbox deployment
• Reviewed build logs for anomalies in anvilbox deployment
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but excluded any p tags}
• Moved connected issue to Merged lower column in ZenHub
• Pushed merge commit to GitHub

Operator (after pushing the merge commit)

• Pushed merge commit to GitLab dev
• Pushed merge commit to GitLab anvildev
• Build passes on GitLab dev
• Reviewed build logs for anomalies on GitLab dev
• Build passes on GitLab anvildev
• Reviewed build logs for anomalies on GitLab anvildev
• Ran _select dev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Deleted PR branch from GitHub
• Deleted PR branch from GitLab dev
• Deleted PR branch from GitLab anvildev

Operator

• At least one hour has passed since anvildev.shared was last deployed
• Ran script/export_inspector_findings.py against anvildev, imported results to Google Sheet and posted screenshot of relevant¹ findings as a comment on the connected issue.
• Added CL item to prod and anvilprod promotion PRs:

• Manually deleted the cerebro, kibana, and gitlab-runner images from anvilprod ( or prod).

• Propagated the deploy:shared, deploy:gitlab, deploy:runner and backup:gitlab labels to the next promotion PRs _{or this PR carries none of these labels}
• Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner and backup:gitlab labels, from the description of this PR to that of the next promotion PRs _{or this PR carries none of these labels}
• PR is assigned to only the system administrator

¹A relevant finding is a high or critical vulnerability in an image
that is used within the security boundary. Images not used within the boundary
are tracked in azul.docker_images under a key starting with _.

System administrator

• No currently reported vulnerability requires immediate attention
• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.90%. Comparing base (9f12a4f) to head (b155323).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #6361   +/-   ##
========================================
  Coverage    84.90%   84.90%           
========================================
  Files          156      156           
  Lines        20692    20692           
========================================
  Hits         17568    17568           
  Misses        3124     3124           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[coveralls]

coverage: 84.915% (-0.005%) from 84.92%
when pulling a578f82 on upgrades/2024-06-24
into de54dcb on develop.

[coveralls]

coverage: 84.92%. remained the same
when pulling b78e2ae on upgrades/2024-06-24
into de54dcb on develop.

[hannes-ucsc]

Unsolicited review. We'll be updating to Docker 17. Note the new blocker for the connected issue. Last commit title should refer to the image it updates, not the command that was run to update it.

[coveralls]

coverage: 84.92%. remained the same
when pulling 0f0f887 on upgrades/2024-06-24
into de54dcb on develop.

[coveralls]

coverage: 84.915% (-0.005%) from 84.92%
when pulling 58ed2d5 on upgrades/2024-06-24
into de54dcb on develop.

[coveralls]

coverage: 84.915% (-0.005%) from 84.92%
when pulling 58ed2d5 on upgrades/2024-06-24
into de54dcb on develop.

[dsotirho-ucsc]

6361_IT_2024-07-01.txt

[hannes-ucsc]

See CL on issue.

[hannes-ucsc]

This is stale already.

[coveralls]

coverage: 84.915% (-0.005%) from 84.92%
when pulling 7e745d2 on upgrades/2024-06-24
into de54dcb on develop.

[hannes-ucsc]

Security design review

• Security design review completed; this PR does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with Terra
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below

[coveralls]

coverage: 84.92%. remained the same
when pulling bf1d569 on upgrades/2024-06-24
into 9f12a4f on develop.

[coveralls]

coverage: 84.92%. remained the same
when pulling b155323 on upgrades/2024-06-24
into 9f12a4f on develop.