✨ New Probe: Memory safety

[balteravishay]

What kind of change does this PR introduce?

This PR provides the foundation for addressing issue #3736 by adding a new probe that checks if the code uses non memory safe practices for the repository languages.
The goal is to automate the detection of as many of the practices that the memory safety SIG provides under the Best Practices - Memory-Safe By Default Languages and the Best Practices - Non Memory-Safe By Default Languages guides.

• PR title follows the guidelines defined in our pull request documentation

What is the current behavior?

Today scorecard does not detect memory safe practices in it's core features or in any of the probes.

What is the new behavior (if this is a feature change)?**

Probe detects the following:

• for golang it detects if the code imports the unsafe package and points to the locations where it is used.

• for c# it detects if the projects allow for unsafe blocks which is a requirement for any project that would use any form of .Net unsafe code, pointer types, and function pointers

• Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

This code change addresses issue #3736 but does not close it (to be discussed)

Special notes for your reviewer

This code change and the implementation of it were discussed in scorecard community calls with @spencerschrock

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

Added independent probe that checks for ecosystem specific non-memory safety practices in the codebase and flags them.

[codecov]

Codecov Report

Attention: Patch coverage is 83.59375% with 21 lines in your changes missing coverage. Please review.

Project coverage is 68.36%. Comparing base (353ed60) to head (0aa68f4).
Report is 124 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4499      +/-   ##
==========================================
+ Coverage   66.80%   68.36%   +1.56%     
==========================================
  Files         230      248      +18     
  Lines       16602    18762    +2160     
==========================================
+ Hits        11091    12827    +1736     
- Misses       4808     5084     +276     
- Partials      703      851     +148     

[balteravishay]

here are some repos to test the code:

go run main.go --repo github.com/microsoft/midi --probes memorysafe --format probe (.net unsafe code)
go run main.go --repo github.com/microsoft/winget-cli --probes memorysafe --format probe (.net safe code)
go run main.go --repo github.com/pkujhd/goloader --probes memorysafe --format probe (go unsafe code)
go run main.go --repo github.com/ossf/scorecard --probes memorysafe --format probe (go safe code)

[spencerschrock]

Hoping to have some time to look at this tomorrow. Quick question about organization:

Added independent probe that checks for ecosystem specific non-memory safety practices

Probes are usually one specific behavior. In this case, the probe is concerned with looking for unsafe usage across ecosystems. I'm not sure if your intention was to use this probe for the go race detector follow up too, but generally different behavior tests get broken up into different probes.

[spencerschrock]

didn't quite go through the tests yet, since the semantics of the probe will probably change through this discussion.

[balteravishay]

Hoping to have some time to look at this tomorrow. Quick question about organization:

Added independent probe that checks for ecosystem specific non-memory safety practices

Probes are usually one specific behavior. In this case, the probe is concerned with looking for unsafe usage across ecosystems. I'm not sure if your intention was to use this probe for the go race detector follow up too, but generally different behavior tests get broken up into different probes.

the intention of this probe is to save the end user the need to know which best practices they are expected to have in the repo, and to give them a kind of a one stop shop for finding out if there's anything they're missing. Our goal was to try and implement the different practices that are suggested in the memory safety guides for memory-safe-by-default and non-memory-safe-by-default guides into this single probe.
The concern with breaking that up for different probes is that user will have to run a bunch of different probes, per ecosystem, per guideline, just to get the overall view.
hope this makes sense.

[spencerschrock]

The concern with breaking that up for different probes is that user will have to run a bunch of different probes, per ecosystem, per guideline, just to get the overall view.

On the other hand, our goal for probes is that consumers are concerned with specific behavior and by having them do one thing it's easier to build policy.

[balteravishay]

The concern with breaking that up for different probes is that user will have to run a bunch of different probes, per ecosystem, per guideline, just to get the overall view.

On the other hand, our goal for probes is that consumers are concerned with specific behavior and by having them do one thing it's easier to build policy.

ok, so just for clarity, for this PR do you want to separate also to two different probes (one for c# and one for go)?

[balteravishay]

@spencerschrock fixed all comments here.

[spencerschrock]

@balteravishay had to tweak this to fix an issue similar to the one in #4531 . Parse errors now return a finding.OutcomeError, similar to Pinned-Dependencies

[balteravishay]

Thank you very much @spencerschrock. Patch looks good to me.