Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(python): use minimum version for pip packages #7348

Merged
merged 11 commits into from
Aug 24, 2024
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions docs/docs/coverage/language/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The following table provides an outline of the features Trivy offers.

| Package manager | File | Transitive dependencies | Dev dependencies | [Dependency graph][dependency-graph] | Position | [Detection Priority][detection-priority] |
|-----------------|------------------|:-----------------------:|:----------------:|:------------------------------------:|:--------:|:----------------------------------------:|
| pip | requirements.txt | - | Include | - | ✓ | - |
| pip | requirements.txt | - | Include | - | ✓ | |
| Pipenv | Pipfile.lock | ✓ | Include | - | ✓ | Not needed |
| Poetry | poetry.lock | ✓ | Exclude | ✓ | - | Not needed |

Expand All @@ -42,8 +42,17 @@ Trivy parses your files generated by package managers in filesystem/repository s
### pip

#### Dependency detection
Trivy only parses [version specifiers](https://packaging.python.org/en/latest/specifications/version-specifiers/#id5) with `==` comparison operator and without `.*`.
To convert unsupported version specifiers - use the `pip freeze` command.
By default, Trivy only parses [version specifiers](https://packaging.python.org/en/latest/specifications/version-specifiers/#id5) with `==` comparison operator and without `.*`.

Using the [--detection-priority comprehensive](#detection-priority) option ensures that the tool establishes a minimum version, which is particularly useful in scenarios where identifying the exact version is challenging.
In such case Trivy parses specifiers `>=`,`~=` and a trailing `.*`.

```
keyring >= 4.1.1 # Minimum version 4.1.1
Mopidy-Dirble ~= 1.1 # Minimum version 1.1
python-gitlab==2.0.* # Minimum version 2.0.0
```
Also, there is a way to convert unsupported version specifiers - use the `pip freeze` command.

```bash
$ cat requirements.txt
Expand Down
27 changes: 23 additions & 4 deletions pkg/dependency/parser/python/pip/parse.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,29 @@ const (
)

type Parser struct {
logger *log.Logger
logger *log.Logger
useMinVersion bool
}

func NewParser() *Parser {
func NewParser(useMinVersion bool) *Parser {
return &Parser{
logger: log.WithPrefix("pip"),
logger: log.WithPrefix("pip"),
useMinVersion: useMinVersion,
}
}
func (p *Parser) splitLine(line string) []string {
separators := []string{"~=", ">=", "=="}
// Without useMinVersion check only `==`
if !p.useMinVersion {
separators = []string{"=="}
}
for _, sep := range separators {
if result := strings.Split(line, sep); len(result) == 2 {
return result
}
}
return nil
}

func (p *Parser) Parse(r xio.ReadSeekerAt) ([]ftypes.Package, []ftypes.Dependency, error) {
// `requirements.txt` can use byte order marks (BOM)
Expand All @@ -53,10 +68,14 @@ func (p *Parser) Parse(r xio.ReadSeekerAt) ([]ftypes.Package, []ftypes.Dependenc
line = rStripByKey(line, commentMarker)
line = rStripByKey(line, endColon)
line = rStripByKey(line, hashMarker)
s := strings.Split(line, "==")

s := p.splitLine(line)
if len(s) != 2 {
continue
}
if p.useMinVersion && strings.Index(s[1], ".*") > -1 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC we can check .* as suffix:

root@5c86285e4f39:/app# pip install django==5.*.*
ERROR: Invalid requirement: 'django==5.*.*': .* suffix can only be used with `==` or `!=` operators
    django==5.*.*
          ~~~~~~^
root@5c86285e4f39:/app# pip install django==4.*.1
ERROR: Invalid requirement: 'django==4.*.1': Expected end or semicolon (after version specifier)
    django==4.*.1
          ~~~~~^
root@5c86285e4f39:/app# pip install django==5.*
Collecting django==5.*
  Downloading Django-5.1-py3-none-any.whl.metadata (4.2 kB)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this check (I mean strings.Index(s[1], ".*") > -1)?
It seems like we can always replace the .* suffix if p.useMinVersion==true.
Or am I missing something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DmitriyLewen ok, removed this check

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, it doesn't work correctly. updating.

s[1] = strings.Replace(s[1], "*", "0", 1)
}

if !isValidName(s[0]) || !isValidVersion(s[1]) {
p.logger.Debug("Invalid package name/version in requirements.txt.", log.String("line", text))
Expand Down
15 changes: 11 additions & 4 deletions pkg/dependency/parser/python/pip/parse_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@ import (

func TestParse(t *testing.T) {
tests := []struct {
name string
filePath string
want []ftypes.Package
name string
filePath string
useMinVersion bool
want []ftypes.Package
}{
{
name: "happy path",
Expand Down Expand Up @@ -66,14 +67,20 @@ func TestParse(t *testing.T) {
filePath: "testdata/requirements_with_templating_engine.txt",
want: nil,
},
{
name: "compatible versions",
filePath: "testdata/requirements_compatible.txt",
useMinVersion: true,
want: requirementsCompatibleVersions,
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
f, err := os.Open(tt.filePath)
require.NoError(t, err)

got, _, err := NewParser().Parse(f)
got, _, err := NewParser(tt.useMinVersion).Parse(f)
require.NoError(t, err)

assert.Equal(t, tt.want, got)
Expand Down
32 changes: 32 additions & 0 deletions pkg/dependency/parser/python/pip/parse_testcase.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,38 @@ package pip
import ftypes "github.com/aquasecurity/trivy/pkg/fanal/types"

var (
requirementsCompatibleVersions = []ftypes.Package{
{
Name: "keyring",
Version: "4.1.1",
Locations: []ftypes.Location{
{
StartLine: 1,
EndLine: 1,
},
},
},
{
Name: "Mopidy-Dirble",
Version: "1.1",
Locations: []ftypes.Location{
{
StartLine: 2,
EndLine: 2,
},
},
},
{
Name: "python-gitlab",
Version: "2.0.0",
Locations: []ftypes.Location{
{
StartLine: 3,
EndLine: 3,
},
},
},
}
requirementsFlask = []ftypes.Package{
{
Name: "click",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
keyring >= 4.1.1 # Minimum version 4.1.1
Mopidy-Dirble ~= 1.1 # Compatible release. Same as >= 1.1, == 1.*
python-gitlab==2.0.*
16 changes: 10 additions & 6 deletions pkg/fanal/analyzer/language/python/pip/pip.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,16 @@ var pythonExecNames = []string{
}

type pipLibraryAnalyzer struct {
logger *log.Logger
metadataParser packaging.Parser
logger *log.Logger
metadataParser packaging.Parser
detectionPriority types.DetectionPriority
}

func newPipLibraryAnalyzer(_ analyzer.AnalyzerOptions) (analyzer.PostAnalyzer, error) {
func newPipLibraryAnalyzer(opts analyzer.AnalyzerOptions) (analyzer.PostAnalyzer, error) {
return pipLibraryAnalyzer{
logger: log.WithPrefix("pip"),
metadataParser: *packaging.NewParser(),
logger: log.WithPrefix("pip"),
metadataParser: *packaging.NewParser(),
detectionPriority: opts.DetectionPriority,
}, nil
}

Expand All @@ -62,8 +64,10 @@ func (a pipLibraryAnalyzer) PostAnalyze(_ context.Context, input analyzer.PostAn
return true
}

useMinVersion := a.detectionPriority == types.PriorityComprehensive

if err = fsutils.WalkDir(input.FS, ".", required, func(pathPath string, d fs.DirEntry, r io.Reader) error {
app, err := language.Parse(types.Pip, pathPath, r, pip.NewParser())
app, err := language.Parse(types.Pip, pathPath, r, pip.NewParser(useMinVersion))
if err != nil {
return xerrors.Errorf("unable to parse requirements.txt: %w", err)
}
Expand Down