Upgrade Alpine image to v18

[kalbfled]

User Story - Business Need

We need to upgrade to the latest version of Alpine 3.18 because there are a number of longstanding security issues with our current version, 3.16, and 3.16's EOL for security fixes is 23 May 2024.

User Story(ies)

As a VA Notify stakeholder,
I want the application image built upon Alpine Linux v3.18 or later
So that the application minimizes security vulnerabilities.

• [ ]Ticket is understood, and QA has been contacted (if the ticket has a QA label).

Additional Info and Resources

There is a new version of urllib3 (v2.0.0) that can't be installed in projects that use botocore because, in the setup.py file, it has the version restriction urllib3>=1.25.4,<1.27. See the current requirement here. Also see the associated ticket in the botocore repository.

That same issue contains a workaround in this comment to utilize the latest version of botocore - author is the maintainer for awscli/boto3:

You can currently install urllib3 2.0.x with boto3/botocore/awscli and it will work as we've unblocked Gentoo and nixpkgs to do. The primary reason that the pin is still in place for general python installations is because urllib3 2.0 has compatibility issues with Amazon Linux 2 and some versions of AWS Lambda. These are detailed in the urllib3 migration guide here.

Because of the way pip works by always choosing to install the latest version of packages, moving the pin for urllib3 will unexpectedly break everyone not pinning urllib3 or using OpenSSL 1.1.1+. That's something we take very seriously with AWS products and try to ensure never happens. We're still working on a plan to smoothly migrate this customer base, but don't have any immediate timeline.

For the time being, users are free to install urllib3 2.x after they've installed boto3/botocore/awscli, and should be able to use each package. Something like this would work in automated environments returning a successful response code. We realize it's not ideal, but is the most tractable solution we have currently that maintains stability:

python -m pip install boto3==x.y.z
python -m pip install urllib3==2.0.4 --no-deps

NOTE at the time of this writing, urllib3 should be pinned to 2.0.6. When picking up this ticket, be sure to check what the latest 2.x.x for urlib3 is. See also the comment added 10/6 regarding a botocore PR recently merged that resolves this issue for python 3.10+.

The notification-api image currently builds from Alpine 3.16. Multiple tickets require upgrading to a newer version of Alpine. See the dependencies section of this ticket.

Upgrading Alpine to v18 to get OpenSSL 3.1.1--is distinct from the task of upgrading the Python "requests" package. The requirements are as follows:

• botocore requires urllib3>=1.25.4,<1.27 
  

• requests 2.31.0 requires urllib3>=1.21.1,<3 
  

• urllib3 >= 2.0.0 requires openssl >= 1.1.1 
  

• Alpine 16 (currently in use) provides openssl 1.1.1; Alpine 18 provides openssl 3.1.1
  

Neither "requests" nor "urllib3" are top level requirements for the Flask app, although 3 lambda functions require "requests". #1241 (closed) upgraded requests to v2.31.0, and "urllib3" to v1.26.x

We need separate tickets to upgrade the "requests" lambda layer used by the lambda functions.

Engineering Checklist

• Upgrade the "FROM" base image in Dockerfile and Dockerfile.local to use Alpine >=3.18
• Update the table here
• Implement the workaround for installing boto3 and urllib3
• Build the image locally

Acceptance Criteria

• The app runs locally and that all unit tests pass
• All QA regression tests pass

QA Considerations

• beat this thing up in regression testing
• Twilio callbacks verified to work without error

Potential Dependencies

Leave blank if n/a

Out of Scope

Leave blank if n/a

[kalbfled]

@k-macmillan @mjones-oddball @cris-oddball Upon further study, I've concluded that this task--upgrading Alpine to v18 to get OpenSSL 3.1.1--is distinct from the task of upgrading the Python "requests" package. The requirements are as follows:

• botocore requires urllib3>=1.25.4,<1.27 (source)
• requests 2.31.0 requires urllib3>=1.21.1,<3 (source)
• urllib3 >= 2.0.0 requires openssl >= 1.1.1 (source)
• Alpine 16 (currently in use) provides openssl 1.1.1; Alpine 18 provides openssl 3.1.1

Neither "requests" nor "urllib3" are top level requirements for the Flask app, although 3 lambda functions require "requests". I expect that "requests" will be upgraded to v2.31.0, and "urllib3" to v1.26.x, as part of #1241, which is in progress. We need separate tickets to upgrade the "requests" lambda layer used by the lambda functions.

I propose that we keep this ticket as-is for upgrading Alpine.

[mjones-oddball]

@kalbfled are you making the requests lambda layer tickets or is Baldwin aware and creating those?

[cris-oddball]

@kalbfled while we are awaiting botocore to support urllib3 >= 2.x.x, can we at least upgrade to Alpine 3.17 instead of jumping to 3.18?

[kalbfled]

@mjones-oddball I created an API and an infra ticket. See #1329.

@cris-oddball Negative. Alpine 3.17 includes OpenSSL v3.0.9, and we can't upgrade to any OpenSSL version above 1.1.1 without upgrading urllib3.

[mjones-oddball]

Hey team! Please add your planning poker estimate with Zenhub @cris-oddball @EvanParish @justaskdavidb2 @k-macmillan @kalbfled @ldraney @nikolai-efimov

[mjones-oddball]

External Blocker: This ticket cannot proceed until the botocore Python package supports urllib3>=2.0.0. See the current requirement here.

[ldraney]

botocore issue link

[k-macmillan]

@cris-oddball You mentioned you have a workaround documented. Please include it here when you are able.

[cris-oddball]

@k-macmillan turns out I had added the workaround directly into the description. :)

[cris-oddball]

@k-macmillan Note that this fix didn't work for everyone. On 10/6/2023, botocore merged a PR that allows urllib3 for python 3.10, so at the end of the day, if the above hack doesn't work, then we should re-focus on upgrading the api to use python 3.10 to get us to alpine 3.18.

[pquentin]

and we can't upgrade to any OpenSSL version above 1.1.1 without upgrading urllib3.

Hello 👋 urllib3 maintainer here. Sorry for jumping in your issue, but urllib3 1.26.x does support OpenSSL 1.1.1 and 3.

[cris-oddball]

Testing Twistlock on this ticket to see which vulnerabilities were cleared.

[cris-oddball]

Cleared over 20 vulnerabilities in twistlock and closed 8 tickets. There are still two remaining vulnerabilities for which I will open new Twistlock tickets.

Diverting testing for Twilio callbacks verified to work without error to the python upgrade ticket and closing this one.