fix(license): stop spliting a long license text

[afdesk]

Description

When we looks for licenses Trivy tries to split information about license through a regex.
but for some cases License field contains a long descriptive text.

This PR adds a detection of a long license text and keep it inside a new field - License, as Dmitriy suggested.

LinceseText field is available for JSON format only. for TABLE format Trivy shows CUSTOM License name instead of a long text.

For tests I use next image:

$ trivy i -d --cache-backend memory --scanners license saisk026/conda-test:v1

Before:

Afrer:

JSON output:

Related issues

• Close don't split licenses from License field from python packaging #5204

Checklist

• I've read the guidelines for contributing to this repository.
• I've followed the conventions in the PR title.
• I've added tests that prove my fix is effective or that my feature works.
• I've updated the documentation with the relevant information (if needed).
• I've added usage information (if the PR introduces new options)
• I've included a "before" and "after" example to the description (if the PR is a user interface change).

[afdesk]

@knqyf263 i think it's ready for review.
please, take a look when you have time. thanks!

[knqyf263]

Is there a way to distinguish between the license name and the license text?

[afdesk]

Is there a way to distinguish between the license name and the license text?

Right now, I'm not sure.
but I'll take a look and will show the cases.

[knqyf263]

I want to show "unknown" for the license text.

[afdesk]

My concern is next.
Python deb packages can contain the license text inside license field:

cat /usr/share/doc/python3.9-minimal/copyright

but if I understand correctly it's a mistake.
the docs: First line (synopsis): an abbreviated name for the license.

[afdesk]

My concern is that there is no a correct way to distinguish between incorrect Permission to use, copy, modify, and distribute this software and its and correct license synopsis TinySCHEME or permissive for gpgv package.

[afdesk]

the same situation is with License field inside *.dist-info.METADATA.

some packages contain a correct license name (eg. Pympler): Apache License, Version 2.0.
some undefined license (eg zope): ZPL 2.1.
some packages contain the license text (eg menuinst):

[afdesk]

@knqyf263 I have an idea. trying

[afdesk]

I want to show "unknown" for the license text.

@knqyf263 Could you confirm that I understand correctly this requirement? thanks

[knqyf263]

I know it's not ideal, but what if checking the length and the number of newlines?

func isLicenseName(license string) bool {
	// Check text length
	if len(license) < 100 {
		return true
	}

	// Count newlines
	if strings.Count(license, "\n") > 3 {
		return false
	}
        ...

[knqyf263]

I want to show "unknown" for the license text.

@knqyf263 Could you confirm that I understand correctly this requirement? thanks

I don't want to show the license text there as it's too long. I thought we would show "UNKNOWN", but we know a license text. We just don't know the short name. How about "Custom"? Then, the license text can be stored in another field.

[afdesk]

I know it's not ideal, but what if checking the length and the number of newlines?

func isLicenseName(license string) bool {
	// Check text length
	if len(license) < 100 {
		return true
	}

	// Count newlines
	if strings.Count(license, "\n") > 3 {
		return false
	}
        ...

@knqyf263 I tried this way, you're right, it's not ideal.

Counting new lines don't affect on the output, because Trivy reads only one line from license in dpkg, python packages also contain a long single line license... so this check is always true.

About check text length. It works for long linceses in python.
but dpkg's licenses are still shown incorrectly.

i thought it's a long text, but actually it's a few first rows of several licenses:

License: Redistribution and use in source and binary forms, with or without
License: By obtaining, using, and/or copying this software and/or its
License: Permission to use, copy, modify, and distribute this software and
License: Redistribution and use in source and binary forms, with or without
License: This software is provided 'as-is', without any express or implied
License: Permission to use, copy, modify, and distribute this software and
License: Permission  is  hereby granted,  free  of charge,  to  any person
License: This software is provided 'as-is', without any express or implied
License: Permission is hereby granted, free of charge, to any person obtaining
   under the terms of the GNU General Public License as published by the
   section entitled ``GNU General Public License''.
License: Permission to use, copy, modify, and distribute this software and its
License: Permission to use, copy, modify, and distribute this software and its
License: This software is provided 'as-is', without any express or implied
License: Permission is hereby granted, free of charge, to any person obtaining
License: Redistribution and use in source and binary forms, with or without
License: This software is provided as-is, without express or implied
License: Permission to use, copy, modify, and distribute this software for any
License: Permission to use, copy, modify, and distribute this software and its
License:
License:  * Permission to use this software in any way is granted without
License: Permission to use, copy, modify, and distribute this software and its

[afdesk]

Right now, I can't see a good solution, but there are several options:

1. We just check text lengths, and don't update output for dpkg licenses, because it's a python's fault.

2. We keep a list of known licenses, and check every splited part.
   if there is no matching, we print a license text.

3. We keep some keywords, and if we can find them inside a string, it'll mean a license text (or a part).
   for example: as-is, redistribution, permission, using, use, @, http etc.

@knqyf263 wdyt?

[afdesk]

I want to show "unknown" for the license text.
I don't want to show the license text there as it's too long. I thought we would show "UNKNOWN", but we know a license text. We just don't know the short name. How about "Custom"? Then, the license text can be stored in another field.

You mean we should print CUSTOM license instead of a long license text for table output, and keep another field (ex licenseText) for JSON, right?
IMHO, It makes sense.
also I'd add a few words from license )

[DmitriyLewen]

i thought it's a long text, but actually it's a few first rows of several licenses:

What if we add one more check for copyright files:
if number of License: * fields more then one - split each line:

• if 2 or more licenses are found in field - it is text license
• if only 1 license is found in field - this is license name

e.g.
line 1 - text, line 2 - license name:

License: Redistribution and use in source and binary forms, with or without
License: BSD-3-Clause

My logic is as follows:
if you use multiple licenses separarted by and/or - you will use only one License: * field.
But if you use multiple License: * fields:

• you will use one license for each License: * field.
  e.g.:

  License: MIT
  License: BSD-3-Clause
  

• in other cases it is license text.

[afdesk]

@DmitriyLewen that's an interesting idea.

there are next cases for perl and python packages:

perl:

License: GPL-1+ or Artistic or Artistic-dist

python3.9:

License: This software is provided 'as-is', without any express or implied

i'm not sure we can separate these cases, but maybe if we also will check string length...

a long string with a few splited licenses is a text.
what is long? I think it about 50 chars.
wdyt?

[DmitriyLewen]

but maybe if we also will check string length...

Yeah. That's what I thought
if there's only one field - we check the length and number of lines.

I think it about 50 chars.

What if use 30 characters + no saved licenses found

[knqyf263]

You mean we should print CUSTOM license instead of a long license text for table output, and keep another field (ex licenseText) for JSON, right?

Exactly

[afdesk]

@knqyf263 @DmitriyLewen
I tried severel ways to separate a long license text from a license name, and the best result for me is a detection by keywords.

There were selected a few obvious words, that can appear inside license texts only.

Please, take a look at this suggestion when you have free time. thanks!

[afdesk]

I tried severel ways to separate a long license text from a license name, and the best result for me is a detection by keywords.

Can you give examples of problems for other methods to save time if we need to come back to this question?

The main problem is to separate a license name and a license text.
Some developers put a full license text inside a license field, where should be a license name (or a license ID).

Checking the length doesn't work because there are correct long lincenses (it's already added to test cases):

Common Development and Distribution License 1.0 (CDDL-1.0)

the number of newlines doesn't work too, because we read only one line from copyright or license files.

the number of spaces doesn't work, because there are too long correct license names (ex CDDL-1.0).

saisk026/conda-test:v1 is a good image for license testing.

[DmitriyLewen]

LGTM

@knqyf263 wdyt about this way?

[DmitriyLewen]

looks like we can skip this change

[afdesk]

this change results the versions of protoc-gen-go for cache/ and common to the same version. it was build automatically

[afdesk]

removed this update for this PR)

[knqyf263]

I tried severel ways to separate a long license text from a license name, and the best result for me is a detection by keywords.

OK, we don't have an easy way. Let's see how it goes.

[afdesk]

I tried severel ways to separate a long license text from a license name, and the best result for me is a detection by keywords.

OK, we don't have an easy way. Let's see how it goes.

Should i fix something to add this PR in 0.55?

[knqyf263]

Should i fix something to add this PR in 0.55?

I'm reviewing the changes now. I'll update you soon.
If we don't make it for v0.55.0, we can include it in v0.55.1.

[knqyf263]

I refactored but realized there was no test. I'm not sure my changes work as expected.
@afdesk Could you add a test for license texts?

[knqyf263]

nit: Since this is "DetectedLicense", Text looks enough.

Suggested change

	LicenseText string
	Text string

[afdesk]

done

[knqyf263]

ditto

[afdesk]

done

[afdesk]

but realized there was no test. I'm not sure my changes work as expected. @afdesk Could you add a test for license texts?

yeh, sure. the test is added