Skip to content

Commit

Permalink
Feature/parser jfrog xray binary scan (DefectDojo#9015)
Browse files Browse the repository at this point in the history 
* new parser Jfrog Xray on Demand Binary Scan

* new parser Jfrog Xray on Demand Binary Scan

* delete blank line at end of file

* rename function

* More sample reports

* Update docs/content/en/integrations/parsers/file/jfrog_xray_on_demand_binary_scan.md

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update docs/content/en/integrations/parsers/file/jfrog_xray_on_demand_binary_scan.md

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update docs/content/en/integrations/parsers/file/jfrog_xray_on_demand_binary_scan.md

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update dojo/settings/settings.dist.py

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update dojo/settings/settings.dist.py

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: kiblik <kiblik@gjh.sk>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: kiblik <kiblik@gjh.sk>

* First round of Improvements

* Drop duplicates in component_id and full_path

* Process per component

* Visual improvements

* Use+clean summary in Title, fix dedup, parse version, drop useless functions

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: kiblik <kiblik@gjh.sk>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: kiblik <kiblik@gjh.sk>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: kiblik <kiblik@gjh.sk>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: kiblik <kiblik@gjh.sk>

* Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py

Co-authored-by: kiblik <kiblik@gjh.sk>

* fix test rename class

* Last Improvements and tests

* capitalization skills

---------

Co-authored-by: Tomas Kubla <tomas@kubla.sk>
Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com>
Co-authored-by: kiblik <kiblik@gjh.sk>
  • Loading branch information
@renejal @kiblik @cneill
4 people authored Dec 2, 2023
1 parent 35808ed commit b42c374
Show file tree
Hide file tree
Showing 10 changed files with 9,710 additions and 0 deletions.
9 changes: 9 additions & 0 deletions docs/content/en/integrations/parsers/file/jfrog_xray_on_demand_binary_scan.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
title: "JFrog Xray On Demand Binary Scan"
toc_hide: true
---
Import the JSON format for the \"JFrog Xray On Demand Binary Scan\" file. Use this importer for Xray version 3.X
--
JFrog file documentation:

https://jfrog.com/help/r/jfrog-cli/on-demand-binary-scan
10 changes: 10 additions & 0 deletions dojo/fixtures/defect_dojo_sample_data.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8620,6 +8620,16 @@
}
},
{
"model": "dojo.test_type",
"pk": 149,
"fields": {
"name": "JFrog Xray On Demand Binary Scan",
"static_tool": false,
"dynamic_tool": false,
"active": true
}
},
{
"model": "dojo.tagulous_product_tags",
"pk": 1,
"fields": {
Expand Down
2 changes: 2 additions & 0 deletions dojo/settings/settings.dist.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1226,6 +1226,7 @@ def saml2_attrib_map_format(dict):
'GitLab Dependency Scanning Report': ['title', 'vulnerability_ids', 'file_path', 'component_name', 'component_version'],
'SpotBugs Scan': ['cwe', 'severity', 'file_path', 'line'],
'JFrog Xray Unified Scan': ['vulnerability_ids', 'file_path', 'component_name', 'component_version'],
'JFrog Xray On Demand Binary Scan': ["title", "component_name", "component_version"],
'Scout Suite Scan': ['file_path', 'vuln_id_from_tool'], # for now we use file_path as there is no attribute for "service"
'AWS Security Hub Scan': ['unique_id_from_tool'],
'Meterian Scan': ['cwe', 'component_name', 'component_version', 'description', 'severity'],
Expand Down Expand Up @@ -1423,6 +1424,7 @@ def saml2_attrib_map_format(dict):
'Checkov Scan': DEDUPE_ALGO_HASH_CODE,
'SpotBugs Scan': DEDUPE_ALGO_HASH_CODE,
'JFrog Xray Unified Scan': DEDUPE_ALGO_HASH_CODE,
'JFrog Xray On Demand Binary Scan': DEDUPE_ALGO_HASH_CODE,
'Scout Suite Scan': DEDUPE_ALGO_HASH_CODE,
'AWS Security Hub Scan': DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL,
'Meterian Scan': DEDUPE_ALGO_HASH_CODE,
Expand Down
Empty file added dojo/tools/jfrog_xray_on_demand_binary_scan/__init__.py
View file
Empty file.
190 changes: 190 additions & 0 deletions dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
import json
import re

from cvss import CVSS3

from dojo.models import Finding


class JFrogXrayOnDemandBinaryScanParser(object):
"""jfrog_xray_scan JSON reports"""

def get_scan_types(self):
return ["JFrog Xray On Demand Binary Scan"]

def get_label_for_scan_types(self, scan_type):
return scan_type

def get_description_for_scan_types(self, scan_type):
return "Import Xray findings in JSON format."

def get_findings(self, json_output, test):
tree = json.load(json_output)
return self.get_items(tree)

def get_items(self, tree):
items = {}
for data in tree:
if "vulnerabilities" in data:
vulnerability_tree = data["vulnerabilities"]

for node in vulnerability_tree:
item_set = get_item_set(node)

for item in item_set:
unique_key = item.title + item.component_name + item.component_version
items[unique_key] = item

return list(items.values())


def get_component_name_version(name):
match = re.match(r"([a-z]+://[a-z\d\.:]+):([a-z\d\.\-]+)", name, re.IGNORECASE)
if match is None:
return name, ""
return match[1], match[2]


def get_severity(vulnerability):
if "severity" in vulnerability:
if vulnerability["severity"] == "Unknown":
severity = "Info"
else:
severity = vulnerability["severity"].title()
else:
severity = "Info"
return severity


def get_references(vulnerability):
if "references" in vulnerability:
ref = ""
references = vulnerability["references"]
for reference in references:
if reference[:2] == "- ":
ref += reference + "\n"
else:
ref += "- " + reference + "\n"
return ref
else:
return None


def get_remediation(extended_information):
remediation = ""
if "remediation" in extended_information:
remediation = "\n\n**Remediation**\n"
remediation += extended_information["remediation"] + "\n"
return remediation


def get_severity_justification(vulnerability):
severity_desc = ""
remediation = ""
extended_information = vulnerability.get("extended_information")
if extended_information:
remediation += get_remediation(extended_information)
if "short_description" in extended_information:
severity_desc += "**Short description**\n"
severity_desc += extended_information["short_description"] + "\n"
if "full_description" in extended_information:
severity_desc += "**Full description**\n"
severity_desc += extended_information["full_description"] + "\n"
if "jfrog_research_severity" in extended_information:
severity_desc += "**JFrog research severity**\n"
severity_desc += extended_information["jfrog_research_severity"] + "\n"
if "jfrog_research_severity_reasons" in extended_information:
severity_desc += "**JFrog research severity reasons**\n"
for item in extended_information["jfrog_research_severity_reasons"]:
severity_desc += item["name"] + "\n" if item.get("name") else ""
severity_desc += item["description"] + "\n" if item.get("description") else ""
severity_desc += "_Is positive:_ " + str(item["is_positive"]).lower() + "\n" if item.get("is_positive") else ""
return severity_desc, remediation


def process_component(component):
mitigation = ""
impact = "**Impact paths**\n\n- "
fixed_versions = component.get("fixed_versions")
if fixed_versions:
mitigation = "**Versions containing a fix:**\n\n- "
mitigation = mitigation + "\n- ".join(fixed_versions)
if "impact_paths" in component:
refs = []
impact_paths_l1 = component["impact_paths"]
for impact_paths_l2 in impact_paths_l1:
for item in impact_paths_l2:
if "component_id" in item:
refs.append(item["component_id"])
if "full_path" in item:
refs.append(item["full_path"])
if refs:
impact += "\n- ".join(sorted(set(refs))) # deduplication
return mitigation, impact


def get_cve(vulnerability):
if "cves" in vulnerability:
cves = vulnerability["cves"]
return cves
return []


def get_vuln_id_from_tool(vulnerability):
if "issue_id" in vulnerability:
return vulnerability["issue_id"]
return None


def clean_title(title):
if title.startswith("Issue summary: "):
title = title[len("Issue summary: "):]
if '\n' in title:
title = title[:title.index('\n')]
return title


def get_item_set(vulnerability):
item_set = []
severity_justification, remediation = get_severity_justification(vulnerability)
severity = get_severity(vulnerability)
references = get_references(vulnerability)
vuln_id_from_tool = get_vuln_id_from_tool(vulnerability)
vulnerability_ids = list()
cvssv3 = None
cvss_v3 = "No CVSS v3 score."
# Some entries have no CVE entries, despite they exist. Example CVE-2017-1000502.
cves = get_cve(vulnerability)
if len(cves) > 0:
for item in cves:
if item.get("cve"):
vulnerability_ids.append(item.get("cve"))
if "cvss_v3_vector" in cves[0]:
cvss_v3 = cves[0]["cvss_v3_vector"]
cvssv3 = CVSS3(cvss_v3).clean_vector()

for component_name, component in vulnerability.get("components", {}).items():
component_name, component_version = get_component_name_version(component_name)
mitigation, impact = process_component(component)

title = clean_title(vulnerability["summary"])
# create the finding object
finding = Finding(
title=title,
severity_justification=severity_justification or None,
severity=severity,
description=(vulnerability["summary"]).strip(),
mitigation=(mitigation + remediation) or None,
component_name=component_name,
component_version=component_version,
impact=impact or None,
references=references or None,
static_finding=True,
dynamic_finding=False,
cvssv3=cvssv3,
vuln_id_from_tool=vuln_id_from_tool,
)
if vulnerability_ids:
finding.unsaved_vulnerability_ids = vulnerability_ids
item_set.append(finding)
return item_set
111 changes: 111 additions & 0 deletions unittests/scans/jfrog_xray_on_demand_binary_scan/many_vulns.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
[
{
"scan_id": "dd8f-4927-5db6-fb188ae8d984",
"vulnerabilities": [
{
"cves": [
{
"cve": "CVE-2017-8923",
"cvss_v2_score": "5.0",
"cvss_v2_vector": "CVSS:2.0/AV:N/AC:L/Au:N/C:N/I:N/A:P",
"cvss_v3_score": "7.5",
"cvss_v3_vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H"
}
],
"summary": "Summary of test",
"severity": "High",
"components": {
"gav://org.yaml:snakeyaml:1.16": {
"fixed_versions": [
"[1.26]"
],
"impact_paths": [
[
{
"component_id": "gav://co.com.test.com"
},
{
"component_id": "gav://co.com.test.com",
"full_path": "lib/snakeyaml-1.16.jar"
}
]
]
}
},
"issue_id": "XRAY-92904",
"references": [
"https://test.com.co"
]
},
{
"cves": [
{
"cve": "CVE-2014-0114",
"cvss_v2_score": "7.5",
"cvss_v2_vector": "CVSS:2.0/AV:N/AC:L/Au:N/C:P/I:P/A:P"
}
],
"summary": "Summary test",
"severity": "High",
"components": {
"gav://test": {
"fixed_versions": [
"[1.9.4]"
],
"impact_paths": [
[
{
"component_id": "gav://co.com.test.test:core:1.0.0-test"
},
{
"component_id": "gav://test",
"full_path": "lib/commons-beanutils-1.9.2.jar"
}
]
]
}
},
"issue_id": "XRAY-55616",
"references": [
"https://test.com.co"
]
},
{
"cves": [
{
"cvss_v2_score": "7.5",
"cvss_v2_vector": "CVSS:2.0/AV:N/AC:L/Au:N/C:P/I:P/A:P"
}
],
"summary": "Summary test",
"severity": "High",
"components": {
"test_item": {
"fixed_versions": [
"[1.2.8.RELEASE]",
"[1.3.1.RELEASE]"
],
"impact_paths": [
[
{
"component_id": "gav://co.com.test.test:core:1.0.0-test"
},
{
"component_id": "gav://test.com.co",
"full_path": "lib/test/libtest"
}
]
]
}
},
"issue_id": "XRAY-79870",
"references": [
"https://test.com.co"
]
}
],
"component_id": "gav://co.com.test.test:core:1.0.0-test",
"package_type": "Maven",
"status": "completed"
}
]
Loading

0 comments on commit b42c374

Please sign in to comment.