scripts: Start of new requirements installer

[aarnq]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

What does this PR do / why do we need this PR?

This aim to develop a lightweight requirements installer that can be used in a container as well as on workstations, using the current PURL format.

The goal is to be able to serve the tests/Dockerfile with the following types:

• deb packages
• helm plugins
• npm packages
• direct bin/tar/zip downloads with checksum verification

As much as possible is to be done in bash alone, debs are always installed first as that may install other package managers, but if needed they are expected to be installed before running the script.

The last type is provided by mapping, which isn't the greatest method, but the one that we kind of already use in the current ansible based approach.

Note that the purl parser does not attempt to be compliant, but tries to be enough for our purposes.

Information to reviewers

Currently I've tested to just run the script against the base requirement file when building a container.

We probably need to have a discussion around the purl types as we are using strange ways to refer to golang packages.
Logically it does not make sense for us to refer to golang packages, as we are not fetching golang packages. It should be github packages for a lot of these as we fetch them from their github releases.

I'll wait with implementing more mappings until we have discussed this.

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change updates CRDs
  • The change updates the config and the schema
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[simonklb]

The reason why I picked the golang PURL type was because vulnerabilities are usually reported for the go modules.

For example:

$ curl -s -d '{"package": {"purl": "pkg:golang/helm.sh/helm"}}' "https://api.osv.dev/v1/query" | jq '.vulns | length'
10
$ curl -s -d '{"package": {"purl": "pkg:github/helm/helm"}}' "https://api.osv.dev/v1/query" | jq '.vulns | length'
0

[Zash]

Suggested change

	checksum="$(sha256sum "${target}")"
	if [[ "${checksum_bin}" == "${checksum%%" "*}" ]]; then
	if sha256sum --check - <<< "${checksum_bin} ${target}" >/dev/null 2>/dev/null; then

Could save a tiny bit of string wrangling, but should otherwise be equivalent.

[aarnq]

The reason why I picked the golang PURL type was because vulnerabilities are usually reported for the go modules.

Sure but if that is going to be a consideration then we should have a at least a plan to take advantage of that, so please create an issue for it else I don't see why we should have roundabout ways to manage it.
And if that is going to be a requirement then many purls are mismatched against OSV.
Would there be any other sources we want to query purls against?

As an alternative, what we have talked about is for the requirement script to be able to download newer versions and at release create a pinned requirements listing. Would it make sense in a such requirements list to write golang purls as an additional alias for that type of use?
Then we could allow the "floating" requirements to have more potential non standard shenanigans to make the fetching easier for us, and the pinned one to be as compliant as we can make it.

[Zash]

Looks promising!

[simonklb]

The reason why I picked the golang PURL type was because vulnerabilities are usually reported for the go modules.

Sure but if that is going to be a consideration then we should have a at least a plan to take advantage of that, so please create an issue for it else I don't see why we should have roundabout ways to manage it.

My understanding was that this issue would cover that.

And if that is going to be a requirement then many purls are mismatched against OSV. Would there be any other sources we want to query purls against?

All the PURLs ideally. But matching CVEs against PURLs isn't a straight forward job, talking from experience from previous engagements... 😄

As an alternative, what we have talked about is for the requirement script to be able to download newer versions and at release create a pinned requirements listing. Would it make sense in a such requirements list to write golang purls as an additional alias for that type of use? Then we could allow the "floating" requirements to have more potential non standard shenanigans to make the fetching easier for us, and the pinned one to be as compliant as we can make it.

Yea I'm not against having a list of "installable" PURLs and then a sublist of aliases/additional resources which can be used to, for example, discover source code for deb packages and vulnerabilities.

[aarnq]

The reason why I picked the golang PURL type was because vulnerabilities are usually reported for the go modules.

Sure but if that is going to be a consideration then we should have a at least a plan to take advantage of that, so please create an issue for it else I don't see why we should have roundabout ways to manage it.

My understanding was that this issue would cover that.

Well, there was nothing there even suggesting that it should be implemented using queries on purls rather than our standard method of subscribing to security notices.
So I want added it.

And if that is going to be a requirement then many purls are mismatched against OSV. Would there be any other sources we want to query purls against?

All the PURLs ideally. But matching CVEs against PURLs isn't a straight forward job, talking from experience from previous engagements... 😄

Note, I'm asking for sources to query against so I can evaluate which purls to use before I create specific mappings that are going to replaced, not which purls to query as I assume we want that for everything. And as you have previous experience, I thought you might have some suggestions. 😉

As an alternative, what we have talked about is for the requirement script to be able to download newer versions and at release create a pinned requirements listing. Would it make sense in a such requirements list to write golang purls as an additional alias for that type of use? Then we could allow the "floating" requirements to have more potential non standard shenanigans to make the fetching easier for us, and the pinned one to be as compliant as we can make it.

Yea I'm not against having a list of "installable" PURLs and then a sublist of aliases/additional resources which can be used to, for example, discover source code for deb packages and vulnerabilities.

Aight, then I'll move towards this because it make more logical sense to me for the requirements installer, and then we can create a framework for increasing our query coverage with mappings later on. 👍

[simonklb]

And if that is going to be a requirement then many purls are mismatched against OSV. Would there be any other sources we want to query purls against?

All the PURLs ideally. But matching CVEs against PURLs isn't a straight forward job, talking from experience from previous engagements... 😄

Note, I'm asking for sources to query against so I can evaluate which purls to use before I create specific mappings that are going to replaced, not which purls to query as I assume we want that for everything. And as you have previous experience, I thought you might have some suggestions. 😉

Sorry I misread it. I am not aware of any other sources that can query PURLs directly so we'll likely have to parse and translate.

Aight, then I'll move towards this because it make more logical sense to me for the requirements installer, and then we can create a framework for increasing our query coverage with mappings later on. 👍

Yea that sounds good. Let's focus on getting installations properly implemented and figure out security scanning later. Having extras/metadata part of the requirements structure doesn't hurt though.