Dockerfile: Temporary adaptation for Ubuntu 20.04

[heliocastro]

Ubuntu 20.04 jumped to python3 mostly, some adaptation are
needed to run basic docker.

This is far to be production ready, as it requires scancode been
compiled from source code instead of releases code but properly
works for Ort been ready to run

TODO:

• Ort requirements test will fail, so it need to be fixe. Commented now
• Dependencies installation inside docker can be done in a smar way,
  with independent installation instead of heavily depends on Ubuntu packages

Signed-off-by: Helio Chissini de Castro helio@kde.org

Please ensure that your pull request adheres to our contribution guidelines. Thank you!

[heliocastro]

This is a WIP.
Works properly, but is far to reach production.
I open this PR as a reference if someone else is planning to work on it

[sschuberth]

This is a WIP.

I'll convert it to a draft PR consequently.

it requires scancode been compiled from source code

Probably, it makes more sense to switch to ScanCode v21.2.9 first, as it has full support for recent Python 3 versions.

[heliocastro]

This is a WIP.

I'll convert it to a draft PR consequently.

it requires scancode been compiled from source code

Probably, it makes more sense to switch to ScanCode v21.2.9 first, as it has full support for recent Python 3 versions.
Yeah, of course !
Because it was released yesterday and for my bad timing i did this three days ago :-)
I will update the PR

[heliocastro]

This is a WIP.

I'll convert it to a draft PR consequently.

it requires scancode been compiled from source code

Probably, it makes more sense to switch to ScanCode v21.2.9 first, as it has full support for recent Python 3 versions.
Yeah, of course !
Because it was released yesterday and for my bad timing i did this three days ago :-)
I will update the PR

Actually not that simple. Scancode new binaries are python 3.6, and bionic is 3.8. I will check that later

[sschuberth]

Actually not that simple. Scancode new binaries are python 3.6, and bionic is 3.8. I will check that later

At least the release notes say "Add support for Python 3.6, 37, 3.8 and 3.9 on Linux". But I'm not sure whether that refers to the package released on PyPI, or only to builds from source. Maybe @pombredanne can explain?

[heliocastro]

Updated:

• Fixed on ScanCode 21.2.9
• Added bash for debug purpose

[pombredanne]

Actually not that simple. Scancode new binaries are python 3.6, and bionic is 3.8. I will check that later

At least the release notes say "Add support for Python 3.6, 37, 3.8 and 3.9 on Linux". But I'm not sure whether that refers to the package released on PyPI, or only to builds from source. Maybe @pombredanne can explain?

The PyPI package that you install with pip runs on 3.6, 37, 3.8 and 3.9.
The app packages released on GitHub with bundled dependencies are designed to run with Python 3.6 only.
https://github.com/nexB/scancode-toolkit/releases/tag/v21.2.9

If you want to install from PyPI you can either use the wheel or source distribution and must use pip install --upgrade scancode-toolkit[full] You may get untested dependency versions. If you want a stable set of deps, same-as-tested in the app release either use the app release or use this:
pip install scancode-toolkit[full] -c https://raw.githubusercontent.com/nexB/scancode-toolkit/v21.2.9/requirements.txt ... this assumes you are using a modern pip and virtualenv in all cases.

[JeroenKnoops]

I needed Scancode for a project as well.

I had some troubles when using the requirements.txt above, so I've installed the full scancode pip package and did a pip freeze and use that for the future installations.

Now its dependency versions are fixed and it's running in python 3.9.

See: https://github.com/philips-software/license-scanner/pull/48/files

[nicorikken]

I tried a similar conversion myself, but then the ort requirements command would give a non-zero exit status. I juist build this Dockerfile version and got a similar result. Let me document it here for reference:

root@a60bb3e7bc7c:/# /opt/ort/bin/./ort requirements
________ _____________________
\_____  \\______   \__    ___/ the OSS Review Toolkit, version DOCKER-SNAPSHOT.
 /   |   \|       _/ |    |
/    |    \    |   \ |    |    Running 'requirements' under Java 11.0.11 on Linux with
\_______  /____|_  / |____|    8 CPUs and a maximum of 3966 MiB of memory.
        \/       \/
Environment variables:
ORT_CONFIG_DIR = /root/.ort/config
ORT_DATA_DIR = /root/.ort
TERM = xterm
JAVA_HOME = /opt/java/openjdk
ANDROID_HOME = /opt/android-sdk
GOPATH = /go

Scanners:
	- Askalono: Requires 'askalono' in version =0.4.3. Tool not found.
	- BoyterLc: Requires 'lc' in version =1.3.1. Tool not found.
	- Licensee: Requires 'licensee' in version =9.13.0. Tool not found.
	* ScanCode: Requires 'scancode' in version =21.2.9. Found version 21.2.9.

PackageManagers:
	* Bower: Requires 'bower' in version >=1.8.8. Found version 1.8.8.
	* Bundler: Requires 'bundle' in version >=1.16.0. Found version 2.1.4.
	* Cargo: Requires 'cargo' in no specific version. Found version 1.46.0.
	* Composer: Requires 'composer' in version >=1.5.0. Found version 1.10.1.
	* Conan: Requires 'conan' in version >=1.18.0. Found version 1.18.0.
	* GoDep: Requires 'dep' in no specific version. Found version 0.5.4.
	* GoMod: Requires 'go' in no specific version. Found version 1.13.4.
	* Npm: Requires 'npm' in version >=5.7.0 <6.15.0. Found version 6.14.2.
	* Pip: Requires 'pip' in no specific version. Found version 20.0.2.
	* Pipenv: Requires 'pipenv' in version >=2018.10.9. Found version 2020.8.13.
	- Pub: Requires 'pub' in version >=2.2.0. Tool not found.
	* Sbt: Requires 'sbt' in version >=0.13.0. Found version 1.3.8.
	* Stack: Requires 'stack' in version >=2.1.1. Found version 2.1.3.
	* Yarn: Requires 'yarn' in version >=1.3.0 <1.23.0. Found version 1.22.4.

VersionControlSystems:
	* Cvs: Requires 'cvs' in no specific version. Found version 1.12.13.
	+ Git: Requires 'git' in version >=2.29.0. Found version 2.25.1.
	* GitRepo: Requires 'repo' in no specific version. Found version 2.15 (launcher).
	* Mercurial: Requires 'hg' in no specific version. Found version 5.3.1.

Other tools:
	* PythonVersion: Requires 'python3' in no specific version. Found version 3.8.10.
	* VirtualEnv: Requires 'virtualenv' in version >=15.1.0 <20.3.0. Found version 20.0.17.

Prefix legend:
	- The tool was not found in the PATH environment.
	+ The tool was found in the PATH environment, but not in the required version.
	* The tool was found in the PATH environment in the required version.

Not all tools were found in their required versions.

When upgrading the Docker image, do we need to consider these requirements as well?

[nicorikken]

With this particular configuration of packages I get errors from ScanCode. For example:

07:58:02.000 [main] INFO  org.ossreviewtoolkit.scanner.LocalScanner - Scanning PyPI::uvicorn:0.14.0' in thread 'main' (45 of 46)
07:58:02.000 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download 'PyPI::uvicorn:0.14.0' sources to '/project/ort/downloads/PyPI/unknown/uvicorn/0.14.0' from VCS...
07:58:02.000 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Using processed VcsInfo(type=Git, url=https://github.com/encode/uvicorn.git, revision=, resolvedRevision=null, path=). Original was VcsInfo(type=, url=, revision=, resolvedRevision=null, path=).
07:58:02.000 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Detected VCS type 'Git' from type name 'Git'.
07:58:02.565 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Adding Git revision '0.14.0' (guessed from package 'uvicorn' and version '0.14.0') as a candidate.
07:58:02.565 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Trying revision candidate '0.14.0' (1 of 1)...
07:58:02.565 [main] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Trying to fetch only revision '0.14.0' with depth limited to 50.
07:58:02.566 [main] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'git fetch --depth 50 origin 0.14.0' in '/project/ort/downloads/PyPI/unknown/uvicorn/0.14.0'...
07:58:04.443 [main] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'git tag 0.14.0 FETCH_HEAD' in '/project/ort/downloads/PyPI/unknown/uvicorn/0.14.0'...
07:58:04.451 [main] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'git checkout 0.14.0' in '/project/ort/downloads/PyPI/unknown/uvicorn/0.14.0'...
07:58:04.471 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Successfully downloaded revision 0.14.0 for package 'PyPI::uvicorn:0.14.0.'.
07:58:04.471 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Finished downloading source code revision '87da6cf4082cd28306bdb78d7d42552d131cc179' to '/project/ort/downloads/PyPI/unknown/uvicorn/0.14.0'.
07:58:04.472 [main] INFO  org.ossreviewtoolkit.scanner.LocalScanner - Running ScannerDetails(name=ScanCode, version=21.2.9, configuration=--copyright --license --info --strip-root --timeout 300 --json-pp) on directory '/project/ort/downloads/PyPI/unknown/uvicorn/0.14.0'.
07:58:04.472 [main] INFO  org.ossreviewtoolkit.scanner.LocalScanner - Archiving files for PyPI::uvicorn:0.14.0.
07:58:04.477 [main] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running '/usr/local/bin/scancode --copyright --license --info --strip-root --timeout 300 --processes 7 /project/ort/downloads/PyPI/unknown/uvicorn/0.14.0 --json-pp /project/ort/native-scan-results/PyPI/unknown/uvicorn/0.14.0/scan-results_ScanCode.json' in '/'...
07:58:05.788 [main] ERROR org.ossreviewtoolkit.scanner.ScanException - Could not scan 'PyPI::uvicorn:0.14.0' (45 of 46): ScanException: Running '/usr/local/bin/scancode --copyright --license --info --strip-root --timeout 300 --processes 7 /project/ort/downloads/PyPI/unknown/uvicorn/0.14.0 --json-pp /project/ort/native-scan-results/PyPI/unknown/uvicorn/0.14.0/scan-results_ScanCode.json' in '/' failed with exit code 2:
/usr/local/lib/python3.8/dist-packages/cluecode/copyrights.py:3095: FutureWarning: Possible set difference at position 3
  remove_tags = re.compile(
/usr/local/lib/python3.8/dist-packages/nltk/tag/sequential.py:540: FutureWarning: Possible nested set at position 7
  self._regexps = [(re.compile(regexp), tag,) for regexp, tag in regexps]
Usage: scancode [OPTIONS] <OUTPUT FORMAT OPTION(s)> <input>...
Try the 'scancode --help' option for help on options and arguments.

Error: Invalid value for '--from-json': Value must be an iterable.

Outside of the container image I can reproduce the error with this version of ScanCode using the provided command. It is resolved with the newest version in Pypi 21.6.7. I'll try it in a new Docker image in the mean time.

[sschuberth]

Superseded by #3902.