[colic] Filter dir paths and softlinks from analysis

[valeriocos]

This PR filters directory paths and softlinks from the colic analysis. This change is needed to prevent failures in scancode-cli (issue #55).

Backend version is now set to 0.5.2

[inishchith]

@valeriocos Thanks for the fix.
I've tested this locally and it works fine!

[valeriocos]

Thank you for reviewing it @inishchith . I launched an execution yesterday and found another error. I'm checking it.

Traceback (most recent call last):
  File "/home/slimbook/Escritorio/sources/graal-libs/scancode-toolkit/etc/scripts/scancli.py", line 73, in <module>
    for s in scan(args):
  File "/home/slimbook/Escritorio/sources/graal-libs/scancode-toolkit/etc/scripts/scancli.py", line 64, in scan
    results = channel.receive()
  File "/usr/local/lib/python3.6/dist-packages/execnet/gateway_base.py", line 728, in receive
    raise self._getremoteerror() or EOFError()
execnet.gateway_base.RemoteError: Traceback (most recent call last):
  File "<string>", line 1063, in executetask
  File "<string>", line 1, in do_exec
  File "<remote exec>", line 53, in <module>
  File "<remote exec>", line 44, in run_scan
  File "/home/slimbook/Escritorio/sources/graal-libs/scancode-toolkit/src/scancode/cli.py", line 813, in run_scan
    raise ScancodeError(msg + '\n' + traceback.format_exc())
ScancodeError: ERROR: failed to collect codebase at: u'/tmp/worktrees/testelkcli/utils/kidash.py'
Traceback (most recent call last):
  File "/home/slimbook/Escritorio/sources/graal-libs/scancode-toolkit/src/scancode/cli.py", line 809, in run_scan
    max_in_memory=max_in_memory
  File "/home/slimbook/Escritorio/sources/graal-libs/scancode-toolkit/src/scancode/resource.py", line 274, in __init__
    self.has_single_resource = bool(self.is_file or not os.listdir(location))
OSError: [Errno 20] No es un directorio: '/tmp/worktrees/testelkcli/utils/kidash.py'


[2019-08-18 06:06:44,309] - Analysis failed at 2fca45dea78c4b254dca24a1623cc8b906dffe2c
Traceback (most recent call last):
  File "/home/slimbook/Escritorio/sources/graal/graal/backends/core/analyzers/scancode.py", line 90, in __analyze_scancode_cli
    msg = subprocess.check_output(cmd_scancli).decode("utf-8")
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['python3', '/home/slimbook/Escritorio/sources/graal-libs/scancode-toolkit/etc/scripts/scancli.py', '/tmp/worktrees/testelkcli/kidash/MANIFEST.in', '/tmp/worktrees/testelkcli/kidash/_version.py', '/tmp/worktrees/testelkcli/setup.py', '/tmp/worktrees/testelkcli/utils/kidash.py']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/slimbook/Escritorio/sources/perceval/perceval/backend.py", line 471, in run
    for item in items:
  File "/home/slimbook/Escritorio/sources/perceval/perceval/backend.py", line 588, in fetch
    raise e
  File "/home/slimbook/Escritorio/sources/perceval/perceval/backend.py", line 582, in fetch
    for item in items:
  File "/home/slimbook/Escritorio/sources/perceval/perceval/backend.py", line 161, in fetch
    for item in self.fetch_items(category, **kwargs):
  File "/home/slimbook/Escritorio/sources/graal/graal/graal.py", line 182, in fetch_items
    raise e
  File "/home/slimbook/Escritorio/sources/graal/graal/graal.py", line 175, in fetch_items
    commit['analysis'] = self._analyze(commit)
  File "/home/slimbook/Escritorio/sources/graal/graal/backends/core/colic.py", line 171, in _analyze
    analysis = self.analyzer.analyze(local_paths)
  File "/home/slimbook/Escritorio/sources/graal/graal/backends/core/colic.py", line 222, in analyze
    analysis = self.analyzer.analyze(**kwargs)
  File "/home/slimbook/Escritorio/sources/graal/graal/backends/core/analyzers/scancode.py", line 127, in analyze
    result = self.__analyze_scancode_cli(kwargs['file_paths'])
  File "/home/slimbook/Escritorio/sources/graal/graal/backends/core/analyzers/scancode.py", line 93, in __analyze_scancode_cli
    e.output.decode("utf-8")))
graal.graal.GraalError: Scancode failed at ['/tmp/worktrees/testelkcli/kidash/MANIFEST.in', '/tmp/worktrees/testelkcli/kidash/_version.py', '/tmp/worktrees/testelkcli/setup.py', '/tmp/worktrees/testelkcli/utils/kidash.py'], [

[valeriocos]

The problem is probably related to the fact that '/tmp/worktrees/testelkcli/utils/kidash.py' is a softlink.

The command below shows that the file is correctly recognized as a "file"

/tmp/worktrees/testelkcli/utils$ test -d kidash.py && echo "dir" || echo "file"

The command below shows that the kidash.py is a softlink:

/tmp/worktrees/testelkcli/utils$ ls -la
total 112
drwxrwxr-x 2 slimbook slimbook  4096 ago 18 11:23 .
drwxrwxr-x 9 slimbook slimbook  4096 ago 18 11:23 ..
-rwxrwxr-x 1 slimbook slimbook  2402 ago 18 11:23 e2k.py
-rwxrwxr-x 1 slimbook slimbook  5835 ago 18 11:23 gelk.py
-rwxrwxr-x 1 slimbook slimbook  4721 ago 18 11:23 get_git_repos.py
-rwxrwxr-x 1 slimbook slimbook  9907 ago 18 11:23 gh2arthur.py
-rwxrwxr-x 1 slimbook slimbook 11870 ago 18 11:23 gh2k.py
-rwxrwxr-x 1 slimbook slimbook   123 ago 18 11:23 gh2k.sh
-rwxrwxr-x 1 slimbook slimbook  2308 ago 18 11:23 google_hits.py
lrwxrwxrwx 1 slimbook slimbook    11 ago 18 11:23 grimoire -> ../grimoire
lrwxrwxrwx 1 slimbook slimbook    15 ago 18 11:23 grimoire_elk -> ../grimoire_elk
lrwxrwxrwx 1 slimbook slimbook    19 ago 18 11:23 kidash.py -> ../kidash/kidash.py
-rwxrwxr-x 1 slimbook slimbook  3282 ago 18 11:23 o2s.py
-rwxrwxr-x 1 slimbook slimbook  3600 ago 18 11:23 ocean.py
-rwxrwxr-x 1 slimbook slimbook  6047 ago 18 11:23 p2o.py
-rwxrwxr-x 1 slimbook slimbook  4343 ago 18 11:23 rest.py
-rw-rw-r-- 1 slimbook slimbook   587 ago 18 11:23 track_gerrit.list
-rwxrwxr-x 1 slimbook slimbook  5275 ago 18 11:23 track_items
-rwxrwxr-x 1 slimbook slimbook  3521 ago 18 11:23 twitter2es.py
-rw-rw-r-- 1 slimbook slimbook   228 ago 18 11:23 twitter.oauth
-rw-rw-r-- 1 slimbook slimbook  7586 ago 18 11:23 twitter_search.py

[valeriocos]

Tested on the full history of ELK, and no error popped up.