Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tern with scancode, exceptions are thrown #964

Closed
chatziparaskewas opened this issue May 10, 2021 · 3 comments · Fixed by #965
Closed

tern with scancode, exceptions are thrown #964

chatziparaskewas opened this issue May 10, 2021 · 3 comments · Fixed by #965
Labels
bug Something went wrong

Comments

@chatziparaskewas
Copy link

Describe the bug
tern with scancode, exceptions are thrown independent of the report type

To Reproduce

  1. Installation of tern and scancode into a virtual environment (~/local/ternenv) as per documentation
  2. Open a new shell
  3. source ~/local/ternenv/bin/activate
  4. run tern (see below); the report type does not matter, I also tried 'json' and 'html', the Docker image is also not the cause (it happens on others as well); an exception is always thrown

Error in terminal

$ tern report -f spdxjson -x scancode -i whereisaaron/kube-syslog-sidecar:latest -o report.json
2021-05-10 16:25:13,519 - DEBUG - __main__ - Starting...
2021-05-10 16:25:13,519 - DEBUG - prep - Setting up...
2021-05-10 16:25:13,568 - DEBUG - run - Starting analysis...
2021-05-10 16:25:13,606 - DEBUG - docker_api - Checking if image "whereisaaron/kube-syslog-sidecar:latest" is available on disk...
2021-05-10 16:25:13,611 - DEBUG - docker_api - Image "whereisaaron/kube-syslog-sidecar:latest" found
2021-05-10 16:25:16,432 - DEBUG - rootfs - Running command: tar -tf /home/xxx/.tern/temp.tar
2021-05-10 16:25:16,450 - DEBUG - rootfs - Running command: tar -x --exclude=.wh.* -f /home/xxx/.tern/temp.tar -C /home/xxx/.tern/temp
2021-05-10 16:25:16,540 - DEBUG - rootfs - Running command: tar -tf /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/layer.tar
2021-05-10 16:25:16,550 - DEBUG - rootfs - Running command: tar -x --exclude=.wh.* -f /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/layer.tar -C /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/contents
2021-05-10 16:25:16,616 - DEBUG - rootfs - Running command: sudo chmod +x /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh
[sudo] password for xxx: 
2021-05-10 16:25:28,952 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/contents
2021-05-10 16:25:29,217 - DEBUG - rootfs - Running command: tar -tf /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/layer.tar
2021-05-10 16:25:29,263 - DEBUG - rootfs - Running command: tar -x --exclude=.wh.* -f /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/layer.tar -C /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/contents
2021-05-10 16:25:29,611 - DEBUG - rootfs - Running command: sudo chmod +x /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh
2021-05-10 16:25:29,673 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/contents
2021-05-10 16:25:48,276 - DEBUG - rootfs - Running command: tar -tf /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/layer.tar
2021-05-10 16:25:48,282 - DEBUG - rootfs - Running command: tar -x --exclude=.wh.* -f /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/layer.tar -C /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/contents
2021-05-10 16:25:48,289 - DEBUG - rootfs - Running command: sudo chmod +x /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh
2021-05-10 16:25:48,353 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/contents
2021-05-10 16:25:48,449 - DEBUG - common - Reading files in filesystem...
2021-05-10 16:25:48,455 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/bin/scancode -ilpcu --quiet --timeout 300 -n 4 --json - /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/contents
2021-05-10 16:26:11,186 - DEBUG - executor - Collecting file data...
2021-05-10 16:26:11,236 - DEBUG - common - Reading files in filesystem...
2021-05-10 16:26:17,191 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/bin/scancode -ilpcu --quiet --timeout 300 -n 4 --json - /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/contents
2021-05-10 16:36:23,199 - DEBUG - executor - Collecting file data...
2021-05-10 16:36:33,999 - DEBUG - common - Reading files in filesystem...
2021-05-10 16:36:34,000 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/bin/scancode -ilpcu --quiet --timeout 300 -n 4 --json - /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/contents
[sudo] password for xxx: 
2021-05-10 16:38:30,825 - DEBUG - executor - Collecting file data...
2021-05-10 16:38:31,896 - DEBUG - generator - Generating SPDX JSON document...
Traceback (most recent call last):
  File "/home/xxx/local/ternenv/bin/tern", line 11, in <module>
    sys.exit(main())
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/__main__.py", line 259, in main
    do_main(args)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/__main__.py", line 108, in do_main
    crun.execute_image(args)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/analyze/default/container/run.py", line 90, in execute_image
    report.report_out(args, full_image)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/report/report.py", line 78, in report_out
    report = generate_report(args, *images)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/report/report.py", line 50, in generate_report
    return generate_format(images, args.report_format, args.print_inclusive)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/report/report.py", line 64, in generate_format
    return mgr.driver.generate(images, print_inclusive)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/formats/spdx/spdxjson/generator.py", line 106, in generate
    report = get_document_dict(image_obj, template)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/formats/spdx/spdxjson/generator.py", line 69, in get_document_dict
    pkgs_dict_list = phelpers.get_packages_list(image_obj, template)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/formats/spdx/spdxjson/package_helpers.py", line 66, in get_packages_list
    package_dicts.append(get_package_dict(package, template))
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/formats/spdx/spdxjson/package_helpers.py", line 41, in get_package_dict
    mapping['PackageLicenseDeclared'] else 'NONE',
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/formats/spdx/spdx_common.py", line 37, in get_license_ref
    return 'LicenseRef-' + get_string_id(license_string)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/formats/spdx/spdx_common.py", line 32, in get_string_id
    return hashlib.sha256(string.encode('utf-8')).hexdigest()[-7:]
AttributeError: 'dict' object has no attribute 'encode'

$ tern report -f html -x scancode -i whereisaaron/kube-syslog-sidecar:latest -o report.html
2021-05-10 16:41:32,765 - DEBUG - __main__ - Starting...
2021-05-10 16:41:32,765 - DEBUG - prep - Setting up...
2021-05-10 16:41:32,793 - DEBUG - run - Starting analysis...
2021-05-10 16:41:32,825 - DEBUG - docker_api - Checking if image "whereisaaron/kube-syslog-sidecar:latest" is available on disk...
2021-05-10 16:41:32,830 - DEBUG - docker_api - Image "whereisaaron/kube-syslog-sidecar:latest" found
2021-05-10 16:41:34,276 - DEBUG - rootfs - Running command: tar -tf /home/xxx/.tern/temp.tar
2021-05-10 16:41:34,282 - DEBUG - rootfs - Running command: tar -x --exclude=.wh.* -f /home/xxx/.tern/temp.tar -C /home/xxx/.tern/temp
2021-05-10 16:41:34,382 - DEBUG - rootfs - Running command: tar -tf /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/layer.tar
2021-05-10 16:41:34,392 - DEBUG - rootfs - Running command: tar -x --exclude=.wh.* -f /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/layer.tar -C /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/contents
2021-05-10 16:41:34,448 - DEBUG - rootfs - Running command: sudo chmod +x /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh
2021-05-10 16:41:34,529 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/contents
2021-05-10 16:41:34,948 - DEBUG - rootfs - Running command: tar -tf /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/layer.tar
2021-05-10 16:41:34,990 - DEBUG - rootfs - Running command: tar -x --exclude=.wh.* -f /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/layer.tar -C /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/contents
2021-05-10 16:41:35,534 - DEBUG - rootfs - Running command: sudo chmod +x /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh
2021-05-10 16:41:35,623 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/contents
2021-05-10 16:41:53,768 - DEBUG - rootfs - Running command: tar -tf /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/layer.tar
2021-05-10 16:41:53,774 - DEBUG - rootfs - Running command: tar -x --exclude=.wh.* -f /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/layer.tar -C /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/contents
2021-05-10 16:41:53,780 - DEBUG - rootfs - Running command: sudo chmod +x /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh
2021-05-10 16:41:53,829 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/tools/fs_hash.sh /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/contents
2021-05-10 16:41:53,907 - DEBUG - common - Reading files in filesystem...
2021-05-10 16:41:53,912 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/bin/scancode -ilpcu --quiet --timeout 300 -n 4 --json - /home/xxx/.tern/temp/b5a95546b3bf73f89269cf29cdf07e1ef408155ab498ea3a2efdeeef63e092a2/contents
2021-05-10 16:42:13,067 - DEBUG - executor - Collecting file data...
2021-05-10 16:42:14,370 - DEBUG - common - Reading files in filesystem...
2021-05-10 16:42:21,357 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/bin/scancode -ilpcu --quiet --timeout 300 -n 4 --json - /home/xxx/.tern/temp/f967f72591675ebfc88ec257661dc67a81e916a393fb8906072fccc7236e9cf8/contents
2021-05-10 16:52:20,441 - DEBUG - executor - Collecting file data...
2021-05-10 16:52:31,411 - DEBUG - common - Reading files in filesystem...
2021-05-10 16:52:31,412 - DEBUG - rootfs - Running command: sudo /home/xxx/local/ternenv/bin/scancode -ilpcu --quiet --timeout 300 -n 4 --json - /home/xxx/.tern/temp/1ec4f758ca078310a4aada7f490f67b7358ae6c79406c3a36680a623d1c8376d/contents
[sudo] password for xxx: 
2021-05-10 16:54:05,761 - DEBUG - executor - Collecting file data...
2021-05-10 16:54:07,292 - DEBUG - generator - Creating HTML report...
Traceback (most recent call last):
  File "/home/xxx/local/ternenv/bin/tern", line 11, in <module>
    sys.exit(main())
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/__main__.py", line 259, in main
    do_main(args)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/__main__.py", line 108, in do_main
    crun.execute_image(args)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/analyze/default/container/run.py", line 90, in execute_image
    report.report_out(args, full_image)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/report/report.py", line 78, in report_out
    report = generate_report(args, *images)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/report/report.py", line 50, in generate_report
    return generate_format(images, args.report_format, args.print_inclusive)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/report/report.py", line 64, in generate_format
    return mgr.driver.generate(images, print_inclusive)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/formats/html/generator.py", line 299, in generate
    report = create_html_report(report_dict, image_obj_list)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/formats/html/generator.py", line 278, in create_html_report
    report = report + '\n' + write_licenses(image_obj_list)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/formats/html/generator.py", line 256, in write_licenses
    licenses = get_licenses_only(image_obj_list)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/report/content.py", line 40, in get_licenses_only
    pkg_licenses = get_layer_packages_licenses(layer)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/report/content.py", line 19, in get_layer_packages_licenses
    package_licenses = get_package_licenses(package)
  File "/home/xxx/local/ternenv/lib64/python3.6/site-packages/tern/report/content.py", line 54, in get_package_licenses
    pkg_licenses.add(package.pkg_license)
TypeError: unhashable type: 'dict'

Expected behavior
Should work, right?

Environment you are running Tern on
Enter all that apply

  • Output of 'tern --version': Tern version 2.5.0
  • Operating System (Linux Distro and version or Mac or Windows): CentOS Linux release 8.3.2011
  • Vagrant file: n/a
  • Container OS: Docker version 20.10.6, build 370c289 / alpine:3.6
  • Python version (3.6 or higher): 3.6.8 (default, Aug 24 2020, 17:57:11)
  • Cloud environment (AWS, Azure, GCP): n/a

Please attach files if they exist

  • tern.log: see above
  • report.*: not created
@nishakm
Copy link
Contributor

nishakm commented May 10, 2021

Yes it should :(. Let me see if I can reproduce it at the tip of the main branch...

@nishakm nishakm added the bug Something went wrong label May 10, 2021
@rnjudge
Copy link
Contributor

rnjudge commented May 10, 2021

I can reproduce the error. It's an issue with the reporting function of Tern not being able to handle whatever data Scancode found because when I run Tern on that image without Scancode I get a SBoM as expected. I will dig in to this.

@rnjudge
Copy link
Contributor

rnjudge commented May 11, 2021 •
edited
Loading

The offending information is actually pip package information that scancode picks up whose license is reported as: {'license': 'MIT', 'classifiers': ['License :: OSI Approved :: MIT License']}. According to the scancode snippet below, a dictionary is expected here for declared_license for python packages:

package = PythonPackage(**common_data)
        declared_license = {}
        if pkginfo.license:
            # TODO: We should make the declared license as it is, this should be updated in scancode to parse a pure string
            declared_license['license'] = pkginfo.license
        if pkginfo.classifiers:
            license_classifiers = []
            other_classifiers = []
            for classifier in pkginfo.classifiers:
                if classifier.startswith('License'):
                    license_classifiers.append(classifier)
                else:
                    other_classifiers.append(classifier)
            declared_license['classifiers'] = license_classifiers
            package.keywords = other_classifiers
        if declared_license:
            package.declared_license = declared_license

This means that in the get_scancode_package() function Tern needs to check for a dict in scancode's declared_license field and handle accordingly for pip packages that might have classifiers attached to them.

 78 def get_scancode_package(package_dict):                                         
 79     '''Given a package dictionary from the scancode results, return a Package   
 80     object with the results'''  
...                                                                                  
 83     package.pkg_license = package_dict['declared_license']

rnjudge added a commit to rnjudge/tern that referenced this issue May 12, 2021
@rnjudge
scancode: filter license from pip pkg classifiers
756dae1 
When scancode detects python package licenses it attaches the license
classifiers to the declared_license dictionary, if applicable. This is a
problem when Tern tries to report the package license by adding it to a
set of licenses in get_package_licenses(), as dictionary objects cannot
be added to sets.

This commit filters out the declared license string from declared
license dictionary containing the classifier values in order to fix
this issue.

Resolves tern-tools#964

Signed-off-by: Rose Judge <rjudge@vmware.com>
@rnjudge rnjudge mentioned this issue May 12, 2021
Merged
nishakm pushed a commit that referenced this issue May 13, 2021
@rnjudge
scancode: filter license from pip pkg classifiers
f6535bb 
When scancode detects python package licenses it attaches the license
classifiers to the declared_license dictionary, if applicable. This is a
problem when Tern tries to report the package license by adding it to a
set of licenses in get_package_licenses(), as dictionary objects cannot
be added to sets.

This commit filters out the declared license string from declared
license dictionary containing the classifier values in order to fix
this issue.

Resolves #964

Signed-off-by: Rose Judge <rjudge@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something went wrong
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

scancode: filter license from pip pkg classifiers rnjudge/tern
3 participants
@nishakm @rnjudge @chatziparaskewas