Separate import and improve operations #525

useless after a530627 Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Improver look strikingly similar to importers with some enhancements. (Eg: It doesn't use an importer_yielder alternative, see issue 501). Overview: Improvers maintain a contract (like Advisory) with improve_runner which is named Inference. Inference class embeds an advisory and a confidence score for that advisory. It is the job of an improver to fetch data to improve from the database (probably using some helper functions) then use whatever means necessary to improve that data sample and return with Inferences. Do note, that Inferences which have already been "imported" by importers would be totally discarded as redundant. Also, in case of two inferences on same data point, the one with highest confidence will be taken into the database. Food for thought: Pssst... Probably Inference class is useless and Advisory class can itself have that confidence score, but then the importers would have to mention that whatever they import have 100% confidence which might be susceptible to typo errors making some importers not mention their confidence thus zeroing on confidence. Anyway, importer and improvers should be different and separated. If not, then we could totally discard the idea of improvers and embed everything in an importer with a confidence score. Well, then, where goes the idea of modularity and keeping things simple ? Also, data coming from an "import"er should always be absolutely correct. This will also ensure that if downstream doesn't want any "improved" data then they don't get our guesses. The whole point of separating importers and improvers is that running improvers could be totally optional and based on downstream taste. Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

This is work in progress, there are a few bugs and a few fixmes as well. Everything will be replaced before the final commit Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Importing Advisory data is now 2 step process. 1) Import Advisory 2) Create relations (default improver) importers are now required to return AdvisoryData instead of Advisory. The data contained inside AdvisoryData is converted into a model named Advisory and then saved into the database. This will be useful later for improvers to work on existing Advisories. All the relationships for the Advisories are generated using a default improver at improvers/default.py. NOTE: There are errors in the UI due to models.py # TODO: Cannot resolve keyword 'resolved_vulnerabilities' into field # make vulnerabilities and resolved_vulnerabilities use the `fix` flag of PackageRelatedVulnerability Knows defects (in current PR): -[ ] UI break (see above) -[ ] might crash in multiple imports / improves -[ ] No improver than default improver is implemented yet -[ ] normalized function of ``AdvisoryData`` has no body -[ ] nginx importer still has remains of set_api etc -[ ] Inference -> AdvisoryData encapsulation -[ ] Duplicated data in database -[ ] ??? Knows defects (to be solved in different PR): -[ ] inconsistent naming - will be resolved in a different PR -[ ] unordered imports Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Earlier Inference used to nest Advisory class, this doesn't make sense logically. Now, the Inference class is of it's own and contains some attributes that could be present in an advisory but are not required. Technical reason behind this decoupling is that the PackageURL in AffectedPackage is NOT supposed to contain version information. This information must be present inside an Inference (that's the whole point). Improvers are required to read AdvisoryData, modify the relevant fields, add inference specific fields (confidence, source) and expand the VersionSpecifier contained inside AffectedPackage package as they deem fit into PackageURLs. Known defects (in current PR): -[ ] UI break (see above) -[ ] might crash in multiple imports / improves -[ ] No improver than default improver is implemented yet -[ ] normalized function of ``AdvisoryData`` has no body -[ ] nginx importer still has remains of set_api etc -[x] Inference -> AdvisoryData encapsulation -[ ] Duplicated data in database -[ ] ??? Knows defects (to be solved in different PR): -[ ] inconsistent naming - will be resolved in a different PR -[ ] unordered imports Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Recent model changes break the UI as now the PackageRelatedVulnerability contains a ``fix`` flag to mark the relationship as a fix. This is leveraged to eliminate multiple columns like patched_package or vulnerable_package. Known defects (in current PR): -[x] UI break -[ ] might crash in multiple imports / improves -[ ] No improver than default improver is implemented yet -[ ] normalized function of ``AdvisoryData`` has no body -[ ] nginx importer still has remains of set_api etc -[x] Inference -> AdvisoryData encapsulation -[ ] Duplicated data in database -[ ] ??? Knows defects (to be solved in different PR): -[ ] inconsistent naming - will be resolved in a different PR -[ ] unordered imports Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

The refactors are based on aboutcode-org#525 (review) - class Inference: order should be the logic order from most important to least important fields - class Improver: docs - data_source.py: Use to_dict() than json() - models: Advisory: docs - improvers/nginx.py: Removed, relevant improvers will now reside inside importer module. In this case, importers/nginx.py - black formatting Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

AffectedPackges now contains all affected versions and one fix version. This is inspired from the design documented at https://docs.google.com/document/d/1sylBGNooKtf220RHQn1I8pZRmqXZQADDQ_TOABrKTpA Under "Format Overview", along the lines of: "affects": { "ranges": [ { "type": string, "repo": string, "introduced": string, "fixed": string } ], "versions": [ string ] }, Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

This is still not perfect because univers is not stable yet. The uncertainty about the structure of version_specifier needs to be resolved. As of now, there are many redundant AffectedPackage objects which would be gone after aboutcode-org/univers#8 is fixed. Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

There must be an affected or fixed purls if there's no vulnerability id. Otherwise, when vulnerability id is present, any (except summery for now) could be present. It does not make sense to neither have vulnerability id nor affected or fixed purls, such relation would be meaningless. Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Updated way: - Each Improver has a method to process a single Advisory model instance such as Improver.get_inferences(self, advisory): -> Inference - The framework then iterates on an Improver-provided QuerySet such as Improver.interesting_advisories that has the Advisories it is interested in. - In the framework, there is an atomic transaction that updates both the Advisory (e.g. date and later a log of improvements with select for update) and whatever is updated or create from the Inference This avoids failing the entire improver when only a single inference is erroneous. Also, the atomic transaction for every advisory and its inferences makes sure that improved_on date of advisory is consistent. Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Git rid of extraneous methods with empty body and commented code. Now as AdvisoryData has all the parameters optional, make sure there's a __post_init__ to assert the constraints. Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

aboutcode-org/univers#10 - parse_version() should accept None and empty strings Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

This is based on PR review done at aboutcode-org#525 (review) Majorly removes commented code, adds some doctest Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

This adds univer from the current branch that support "vers". See aboutcode-org/univers#12 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

All of this comes from the review done at aboutcode-org#525 (review) Co-authored-by: Philippe Ombredanne <pombredanne@gmail.com> Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

This commit adopts the new univers, this also requires to reset all the migrations as migration 0003 was using a univers method which is no longer available. There are also a few refactors based on aboutcode-org#525 (review) Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

because why not ? Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Rewrite following improvers: DefaultImprover Plus adjust the improver framework Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

VersionRange is always a flat list of constraints Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

An alias is a unique vulnerability identifier in some database, such as the NVD, PYSEC, CVE or similar. These databases guarantee that these identifiers are unique within their namespace. An alias may also be used as a Reference. But in contrast with some Reference may not be an identifier for a single vulnerability, for instance, security advisories such as Debian security advisory reference various vulnerabilities. Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

from_dict is a factory, should be a classmethod Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

The entire view is messed up. Will probably need a rewrite Co-authored-by: Philippe Ombredanne <pombredanne@gmail.com> Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Fixes are in accordance with the final review for this PR done at : aboutcode-org#525 (review) Signed-off-by: Hritik Vijay <hritikxx8@gmail.com> Co-authored-by: Philippe Ombredanne <pombredanne@gmail.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Separate import and improve operations #525

Separate import and improve operations #525

Commits on Jan 25, 2022