Manage vulnerability sources in database #152

haikoschol · 2020-03-01T14:31:03Z

This is an implementation of #123.

The change adds a new model Importer that stores metadata as well as the name of a class (a vulnerabilities.data_source.DataSource) that fetches and transforms vulnerability data into a common, DB schema agnostic format. This format is encoded in the two data classes vulnerabilities.data_source.Package and vulnerabilities.data_source.VulnerabilityInfo.

Running an Importer and storing the results in the database is done by ImportRunner.

The unused and redundant model PackageReference has been removed.

Another minor, but overdue change is renaming the scraper module to importers.

@reviewers:

It might be helpful to start with looking at how the import management command uses the above described classes and then jump into the rest of the code from there.

vulnerabilities/models.py

sbs2001 · 2020-03-17T12:26:01Z

vulnerabilities/import_runner.py

+        if vuln_info.cve_id:
+            vuln, created = models.Vulnerability.objects.get_or_create(cve_id=vuln_info.cve_id)
+            if created:
+                vuln.summary = vuln_info.summary


We need some way to determine, which summary is better and then decide to update the existing summary or leave it as it is.

I'm not optimistic there is one. But even if there was a way, if all we have is the summary, we can't detect that a new vulnerability is actually an update to an existing vulnerability.

I was thinking of moving summary from Vulnerability model to the ImpactedPackage model . So the context won't be lost.

In theory it makes sense that the same vulnerability could be described in different terms, depending on the affected package. But in practice, do any of the source we ingest so far do this? Or would we end up copying the same description to every ImpactedPackage? That wouldn't really be an issue though.

Fundamentally the questions are:

What is the definition of "vulnerability"?

What is the minimum number of attributes (and what are those) for identifying a vulnerability?

vulnerabilities/import_runner.py

vulnerabilities/data_source.py

haikoschol · 2020-05-23T14:48:55Z

Don't we need a migration script to have an Importer for safety_db , since you have implemented new importer for it ?

We do and it seems I missed that. Good catch!

Bump the Python 3.6 to Python 3.8 in the README :)

Also good catch! :)

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

This change introduces the model Importer, which contains metadata and a pointer to the implementation of the importers DataSource subclass. This subclass knows how to query whatever storage mechanism is used in that case (e.g. git repository with yaml files, a single OVAL XML file, HTTP API returning JSON, etc.) Additionally it defines an API and data model that DataSource subclasses need to provide to ImportRunner. ImportRunner leverages that API and data model to efficiently store vulnerability information in the database. Existing importers have not been migrated yet (only renamed from "scraper"). Signed-off-by: Haiko Schol <hs@haikoschol.com>

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Having separate methods added_files() and updated_files() is not ideal since the same file can be returned by both methods. When a file has been added and then later modified within the window of time considered by these methods, the file should only be reported as an added file. Therefore this commit combines the methods added_files() and updated_files() into file_changes(), which returns a tuple of sets for added and updated files, respectively. It also fixes a time-math issue; commit timestamps are converted to UTC before being compared to the cutoff timestamp (that is always in UTC). Signed-off-by: Haiko Schol <hs@haikoschol.com>

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Reduce the amount of converting between PackageURL and string to the bare minimum. Add/improve some type annotations and reduce API surface area. Signed-off-by: Haiko Schol <hs@haikoschol.com>

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Many derived classes need a Unix timestamp with the cutoff date as well as a method to split a list of advisories into batches. Signed-off-by: Haiko Schol <hs@haikoschol.com>

Signed-off-by: Haiko Schol <hs@haikoschol.com>

haikoschol added the WIP Work in progress PRs, tickets, etc. label Mar 1, 2020

haikoschol requested a review from pombredanne March 1, 2020 14:31

haikoschol mentioned this pull request Mar 5, 2020

Duplicates in api response after every data dump #156

Closed

sbs2001 reviewed Mar 6, 2020

View reviewed changes

vulnerabilities/models.py Outdated Show resolved Hide resolved

haikoschol force-pushed the 123-vulnsrc-in-db branch 4 times, most recently from 7a7295c to 6404723 Compare March 15, 2020 14:01

haikoschol force-pushed the 123-vulnsrc-in-db branch 2 times, most recently from 4e76c42 to 08ef849 Compare March 15, 2020 20:14

haikoschol linked an issue Mar 15, 2020 that may be closed by this pull request

Data Ingestion Scenarios #164

Closed

sbs2001 reviewed Mar 17, 2020

View reviewed changes

haikoschol force-pushed the 123-vulnsrc-in-db branch 5 times, most recently from c9e33a2 to 9c54a5d Compare April 4, 2020 15:38

haikoschol mentioned this pull request Apr 4, 2020

Use and store hashes of all imported data #169

Closed

haikoschol force-pushed the 123-vulnsrc-in-db branch 2 times, most recently from 7e322b7 to 63e607d Compare April 10, 2020 21:38

sbs2001 reviewed Apr 11, 2020

View reviewed changes

vulnerabilities/import_runner.py Show resolved Hide resolved

haikoschol force-pushed the 123-vulnsrc-in-db branch 4 times, most recently from 87ea90a to 0eed1b5 Compare April 17, 2020 19:46

sbs2001 reviewed Apr 18, 2020

View reviewed changes

vulnerabilities/data_source.py Outdated Show resolved Hide resolved

haikoschol force-pushed the 123-vulnsrc-in-db branch 3 times, most recently from 16e4afb to 6c4c5a1 Compare April 21, 2020 17:46

haikoschol force-pushed the 123-vulnsrc-in-db branch from 6c4c5a1 to 38ad8fe Compare May 2, 2020 10:29

haikoschol removed the WIP Work in progress PRs, tickets, etc. label May 23, 2020

haikoschol force-pushed the 123-vulnsrc-in-db branch from a42de02 to 88fbe9f Compare May 23, 2020 14:56

sbs2001 and others added 18 commits May 23, 2020 16:58

Add importer for safetydb

4657a8a

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Add base class for git-based data sources

5b64977

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Fix typo

2c48240

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Remove call to deleted method DataSource.apply_config()

2e208c4

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Return absolute paths from GitDataSource.file_changes()

b4a945a

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Make Advisory type hashable

d0a25ac

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Implement Rust importer based on GitDataSource

2672d5e

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Implement Alpine importer based on GitDataSource

33fa094

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Clean up ImportRunner and DataSource

ee9e235

Reduce the amount of converting between PackageURL and string to the bare minimum. Add/improve some type annotations and reduce API surface area. Signed-off-by: Haiko Schol <hs@haikoschol.com>

Uncomment temporarily disabled tests

740152a

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Implement Arch Linux importer based on DataSource

8b3277e

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Implement Debian importer based on DataSource

b4008a5

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Implement SafetyDB importer based on DataSource

8d4592b

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Add often required functionality to DataSource

1e9a1d3

Many derived classes need a Unix timestamp with the cutoff date as well as a method to split a list of advisories into batches. Signed-off-by: Haiko Schol <hs@haikoschol.com>

Harmonize naming of package version caching class

3f52a1b

Signed-off-by: Haiko Schol <hs@haikoschol.com>

Implement NPM importer based on DataSource

f342fee

Signed-off-by: Haiko Schol <hs@haikoschol.com>

haikoschol force-pushed the 123-vulnsrc-in-db branch from 88fbe9f to 06c6314 Compare May 23, 2020 15:26

Update required Python version in README and Travis

42c99fb

Signed-off-by: Haiko Schol <hs@haikoschol.com>

haikoschol force-pushed the 123-vulnsrc-in-db branch from 06c6314 to 42c99fb Compare May 23, 2020 15:29

Fix lint

b74bbe4

Signed-off-by: Haiko Schol <hs@haikoschol.com>

haikoschol merged commit 51bb8ce into aboutcode-org:develop May 23, 2020

haikoschol mentioned this pull request May 26, 2020

Manage vulnerability sources in database #123

Closed

sbs2001 mentioned this pull request May 30, 2020

Debian vulnerabilities are by default looked for 'jessie' release #139

Closed

sbs2001 mentioned this pull request Apr 26, 2021

Replace objects.create with objects.update_or_create. #40

Closed

haikoschol deleted the 123-vulnsrc-in-db branch April 10, 2023 00:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Manage vulnerability sources in database #152

Manage vulnerability sources in database #152

haikoschol commented Mar 1, 2020 •

edited

Loading

sbs2001 Mar 17, 2020 •

edited

Loading

haikoschol Mar 22, 2020

sbs2001 Mar 23, 2020

haikoschol Mar 29, 2020

haikoschol commented May 23, 2020

Manage vulnerability sources in database #152

Manage vulnerability sources in database #152

Conversation

haikoschol commented Mar 1, 2020 • edited Loading

sbs2001 Mar 17, 2020 • edited Loading

Choose a reason for hiding this comment

haikoschol Mar 22, 2020

Choose a reason for hiding this comment

sbs2001 Mar 23, 2020

Choose a reason for hiding this comment

haikoschol Mar 29, 2020

Choose a reason for hiding this comment

haikoschol commented May 23, 2020

haikoschol commented Mar 1, 2020 •

edited

Loading

sbs2001 Mar 17, 2020 •

edited

Loading