Merge branch 'develop'

fasten-project · Nov 3, 2020 · 8c5a8ef · 8c5a8ef
2 parents 28b3e9f + a0679d2
commit 8c5a8ef
Show file tree

Hide file tree

Showing 46 changed files with 482 additions and 137 deletions.
diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml
@@ -36,14 +36,15 @@ jobs:
       run: |
         flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
         flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-    - name: Run unit tests
-      run: |
-        pytest rapidplugin/tests
 
     - name: Build docker image
       run: |
         docker build -t rapidplugin-test .
 
+    - name: Run unit tests
+      run: |
+        docker run --entrypoint python rapidplugin-test -m pytest rapidplugin/tests
+
     - name: Setup integration environment
       run: |
         docker-compose -f integration_tests/kafka-compose.yml up -d

diff --git a/Dockerfile b/Dockerfile
@@ -15,6 +15,9 @@ RUN useradd -m plugin
 
 USER plugin
 
+RUN git config --global user.email "research@softwareimprovementgroup.com" \
+ && git config --global user.name "Software Improvement Group Research"
+
 WORKDIR /home/plugin
 
 COPY --chown=plugin rapidplugin rapidplugin/

diff --git a/README.md b/README.md
@@ -9,3 +9,206 @@
 # RAPID
 
 RAPID is the quality analysis application that SIG developes for the FASTEN project in WP3.
+
+# Components
+
+## RapidPlugin
+
+RapidPlugin is a FASTEN plugin that 
+generates code complexity data for the `product`.
+
+The plugin consumes messages in the Kafka topics, 
+generates code complexity data using `Lizard`, and 
+produces Kafka topics with complexity data at the `callable` level.
+
+### Input Kafka topics
+
+The plugin will trigger different pipelines based on the `forge`, 
+so the field `forge` is a mandatory in any incoming messages.
+The currently supported forges are "mvn", "debian", and "PyPI". 
+The plugin will raise an exception if the `forge` in the message is not supported or empty.
+
+#### Maven
+The default topic to consume: `fasten.RepoCloner.out`
+
+An example message:
+
+```json
+{
+  "input": {},
+  "host": "fasten-repo-cloner-56dcf76495-bn4c2",
+  "created_at": 1602739158,
+  "plugin_name": "RepoCloner",
+  "payload": {
+    "repoUrl": "",
+    "date": 1291905586,
+    "forge": "mvn",
+    "groupId": "fasten-project",
+    "artifactId": "fasten",
+    "version": "1.0.0",
+    "sourcesUrl": "http://fasten-project/fasten/fasten-1.0.0-sources.jar",
+    "repoPath": "/mnt/fasten/repos/f/fasten-project/fasten",
+    "repoType": "git",
+    "commitTag": "v1.0.0"
+  }
+}
+```
+The message should have all the information to identify a unique `product`.
+For **Maven**, the fields `groupId`, `artifactId`, and `version` 
+should not be empty. 
+Missing any of these fields will cause exceptions in message consuming.
+
+The message should have at least one way to point out the link to the source code.
+- `sourcesUrl`
+
+This field is the most reliable pointer to the versioned source code of a `product`. 
+
+If `sourcesUrl` presents and is non-empty, the plugin will download the source code from the url specified in `sourcesUrl`.
+
+If `sourcesUrl` does not present or is empty, the plugin will try the other sources to get the source code.
+
+- `repoPath`
+
+If `repoPath` is not empty, the repository of the `product` has been cloned to the FASTEN server. 
+The plugin will try to check out the right version of the source code if both `repoType` and `commitTag` are non-empty.
+The currently supported`repoType` are "git", "svn", and "hg". 
+The plugin will raise an exception if the `repoType` in the message is not supported.
+
+If none of the above efforts succeed, the plugin will raise an exception 
+specifying that it cannot get the source code.
+
+#### PyPI
+The current topic to consume: `fasten.pycg.with_sources.out`
+
+An example message:
+
+```json
+{
+  "input": {},
+  "plugin_name": "PyCG",
+  "plugin_version": "0.0.1",
+  "created_at": "1596576724",
+  "payload": {
+    "product": "gud",
+    "forge": "PyPI",
+    "generator": "PyCG",
+    "depset": [],
+    "version": "1.0.10",
+    "timestamp": "1561421985",
+    "modules": {
+    },
+    "graph":{},
+    "cha": {},
+    "metadata": {},
+    "sourcePath": "/mnt/fasten/pypi-test/final/sources/g/gud/1.0.10"
+  }
+}
+```
+
+The message should have all the information to identify a unique `product`.
+For **PyPI**, the fields `product`, and `version` should not be empty. 
+Missing any of these fields will cause exceptions in message consuming.
+
+If `sourcePath` is empty, the plugin will raise an exception.
+
+#### Debian 
+The current topic to consume: `fasten.debian.cg.2`
+
+An example message:
+
+```json
+{
+  "plugin_name": "CScoutKafkaPlugin",
+  "plugin_version": "0.0.1",
+  "input": {
+    "package": "sed",
+    "version": "4.7-1",
+    "arch": "amd64",
+    "release": "buster",
+    "source": "sed",
+    "source_version": "4.7-1",
+    "date": ""
+  },
+  "created_at": "1600861444.064117",
+  "payload": {
+    "forge": "debian",
+    "release": "",
+    "product": "sed",
+    "version": "4.7-1",
+    "source": "sed",
+    "architecture": "amd64",
+    "generator": "cscout",
+    "timestamp": "1545470644",
+    "depset": [],
+    "build_depset": [],
+    "undeclared_depset": [],
+    "graph": {},
+    "functions": {},
+    "profiling_data": {},
+    "sourcePath": "/mnt/fasten/debian/sources/s/sed/4.7-1"
+  }
+}
+```
+Similar to **PyPI**, the fields `product`, and `version` should not be empty. 
+Missing any of these fields will cause exceptions in message consuming.
+
+If `sourcePath` is empty, the plugin will raise an exception.
+
+### Output Kafka topics
+
+The field `input` in the output topic is used for tracking upstreaming plugins and 
+usually copies the whole `payload` from the consumed message. 
+To avoid potential large message in the output topics, 
+the plugin will tailor the `payload` of consumed message. 
+The content of the following fields will be tailored. 
+
+**PyPI**: `fasten.pycg.with_sources.out`
+- `depset`
+- `cha`
+- `graph`
+- `modules`
+
+**Debian**: `fasten.debian.cg.2`
+- `depset`
+- `build_depset`
+- `undeclared_depset`
+- `graph`
+- `functions`
+
+#### Output topic
+The default topic to produce: `fasten.RapidPlugin.callable.out`
+
+An example message:
+```json
+{
+  "plugin_name": "RapidPlugin",
+  "plugin_version": "0.0.1",
+  "input": {},
+  "created_at": "1595434993",
+  "payload": {
+    "analyzer_name": "Lizard",
+    "analyzer_version": "1.17.7",
+    "analysis_timestamp": "1596455923",
+    "product": "fasten-project:fasten",
+    "version": "1.0.0",
+    "forge": "mvn",
+    "language": "java",
+    "filepath": "/fasten/core/server.java",
+    "name": "callable",
+    "long_name": "callable(int i)",
+    "start_line": 33,
+    "end_line": 42,
+    "metrics": {
+      "nloc": 10,
+      "complexity": 5,  
+      "...": 0
+    }
+  }
+}
+```
+
+#### Log topic
+The default topic to produce: `fasten.RapidPlugin.callable.log`
+
+#### Error topic
+The default topic to produce: `fasten.RapidPlugin.callable.err`
diff --git a/integration_tests/resources/fasten.RepoCloner.out-testdata.txt b/integration_tests/resources/fasten.RepoCloner.out-testdata.txt
diff --git a/integration_tests/test_plugin.py b/integration_tests/test_plugin.py
@@ -61,10 +61,8 @@ def mock_err():
 @pytest.fixture()
 def plugin_run(mock_in, mock_out, mock_log, mock_err,
                in_message):
-    fixed_in_message = fix_sourcePath(in_message,
-                                      '/home/plugin/rapidplugin/tests/resources')
-    mock_in.emit_message(mock_in.produce_topic, fixed_in_message,
-                         "[TEST]", fixed_in_message)
+    mock_in.emit_message(mock_in.produce_topic, in_message,
+                         "[TEST]", in_message)
     sleep(2)
     mock_out.consume_messages()
     mock_log.consume_messages()
@@ -89,7 +87,7 @@ def plugin_run(mock_in, mock_out, mock_log, mock_err,
     #     "artifactId": "m1",
     #     "version": "1.0.0",
     #     "sourcesUrl": "",
-    #     "repoPath": "maven/git/m1",
+    #     "repoPath": "/home/plugin/rapidplugin/tests/resources/maven/git/m1",
     #     "repoType": "git",
     #     "commitTag": "1.0.0"
     # },
@@ -99,31 +97,31 @@ def plugin_run(mock_in, mock_out, mock_log, mock_err,
     #     "artifactId": "m2",
     #     "version": "1.0.0",
     #     "sourcesUrl": "",
-    #     "repoPath": "maven/svn/m2",
+    #     "repoPath": "/home/plugin/rapidplugin/tests/resources/maven/svn/m2",
     #     "repoType": "svn",
     #     "commitTag": "1.0.0"
     # },
-    # {
-    #     "forge": "mvn",
-    #     "groupId": "test-mvn",
-    #     "artifactId": "m3",
-    #     "version": "1.0.0",
-    #     "sourcesUrl": "",
-    #     "repoPath": "maven/hg/m3",
-    #     "repoType": "hg",
-    #     "commitTag": "1.0.0"
-    # },
+    {
+        "forge": "mvn",
+        "groupId": "test-mvn",
+        "artifactId": "m3",
+        "version": "1.0.0",
+        "sourcesUrl": "",
+        "repoPath": "/home/plugin/rapidplugin/tests/resources/maven/hg/m3",
+        "repoType": "hg",
+        "commitTag": "1.0.0"
+    },
     {
         "forge": "debian",
         "product": "d1",
         "version": "1.0.0",
-        "sourcePath": "debian/d1"
+        "sourcePath": "/home/plugin/rapidplugin/tests/resources/debian/d1"
     },
     {
         "forge": "PyPI",
         "product": "p1",
         "version": "1.0.0",
-        "sourcePath": "pypi/p1"
+        "sourcePath": "/home/plugin/rapidplugin/tests/resources/pypi/p1"
     }])
 def test_successes(plugin_run, in_message):
     out, log, err = plugin_run
@@ -145,6 +143,26 @@ def test_successes(plugin_run, in_message):
         "forge": "PyPI",
         "product": "p1",
         "version": "1.0.0"
+    },
+    {
+        "forge": "mvn",
+        "groupId": "test-mvn",
+        "artifactId": "m2",
+        "version": "1.0.0",
+        "sourcesUrl": "",
+        "repoPath": "maven/svn/m2",
+        "repoType": "svn",
+        "commitTag": "1.0.0"
+    },
+    {
+        "forge": "mvn",
+        "groupId": "test-mvn",
+        "artifactId": "m3",
+        "version": "1.0.1",
+        "sourcesUrl": "",
+        "repoPath": "/home/plugin/rapidplugin/tests/resources/maven/hg/m3",
+        "repoType": "hg",
+        "commitTag": "1.0.1"
     }])
 def test_failures(plugin_run, in_message):
     out, log, err = plugin_run

diff --git a/rapidplugin/analysis/lizard_analyzer.py b/rapidplugin/analysis/lizard_analyzer.py
@@ -16,6 +16,8 @@
 import os
 import logging
 import datetime
+from pathlib import Path
+
 import lizard
 
 from rapidplugin.domain.package import Package, File, Function
@@ -37,8 +39,8 @@ def analyze(self, payload):
         forge = payload['forge']
         product = payload['groupId'] + ":" + payload['artifactId'] if forge == "mvn" else payload['product']
         version = payload['version']
-        path = self.get_source_path(payload)
-        package = LizardPackage(forge, product, version, path)
+        with MavenUtils.get_source_path(payload, self.base_dir) as path:
+            package = LizardPackage(forge, product, version, str(path))
         metadata = package.metadata()
         for function in package.functions():
             m = {}
@@ -49,39 +51,6 @@ def analyze(self, payload):
             logger.debug("callable: {}".format(m) + '\n')
         return out_payloads
 
-    def get_source_path(self, payload):
-        """
-        TODO: consider moving this to a utility class.
-        For maven, the order to get source code path from different sources:
-        [x] 1. if *-sources.jar is valid, download,
-               uncompress and return the path to the source code
-        [x] 2. else if repoPath is not empty, and
-        [x]    2.1 if commit tag is valid, checkout based on tag and return the path
-        [ ]    2.2 if needed, checkout based on the release date.
-        [ ] 3. else return null
-        """
-        if payload['forge'] == "mvn":
-            if 'sourcesUrl' in payload:
-                sources_url = payload['sourcesUrl']
-                return MavenUtils.download_jar(sources_url, self.base_dir)
-            else:
-                if 'repoPath' in payload and 'commitTag' in payload and 'repoType' in payload:
-                    repo_path = payload['repoPath']
-                    repo_type = payload['repoType']
-                    commit_tag = payload['commitTag']
-                    return MavenUtils.checkout_version(repo_path, repo_type, commit_tag)
-        else:
-            source_path = payload['sourcePath']
-            assert os.path.isabs(source_path), "sourcePath is not an absolute path!"
-            return source_path
-
-    def clean_up(self):
-        '''
-        TODO
-        '''
-        # if os.path.exists(self.base_dir):
-        #     shutil.rmtree(self.base_dir)
-
 
 class LizardPackage(Package):