starting directory for pktvisor automated tests

automated_tests/.gitignore

@@ -0,0 +1,7 @@
+test_config.ini
+reports/
+behave_test/
+/site
+/env_tests
+__pycache__/
+doc_generator.py

automated_tests/README.md

@@ -0,0 +1,74 @@
+# Pktvisor Tests
+This directory contains automated tests for pktvisor
+
+
+This directory is organized as described below:
+
+
+```
+python-test
+├── README.md
+├── requirements.txt
+├── docs
+└── features
+|   ├── steps
+|   └── .feature files
+
+```
+
+- Inside the "docs" folder is the test scenarios' documentation
+- Test features are inside the "features" folder and consist of .features files. This is the gherkin language description of the scenarios you will see in the execution terminal
+- The python programming of each step of the scenarios is inside the "steps" subfolder, inside "features"
+
+
+<br>
+
+Here's what you'll need to do in order to run these tests:
+- Setup your python environment
+- Run behave
+
+## Setup your Python environment
+Create a virtual environment: `python3 -m venv name_of_virtualenv`
+
+Activate your virtual environment: `source name_of_virtualenv/bin/activate`
+
+Install the required libraries: `pip install -r requirements.txt`
+
+
+## Run behave
+From the root of the repository simply run `behave`, optionally passing the feature file as follows:
+
+```sh
+$ behave features/pktvisor.feature
+```
+
+Output:
+
+```
+Feature: pktvisor tests # features/pktvisor.feature:1
+
+  Scenario: pktvisor bootstrap                         # features/pktvisor.feature:3
+    When run pktvisor instance on port default         # features/steps/pktvisor.py:33 0.150s
+    Then the pktvisor container status must be running # features/steps/pktvisor.py:39 0.007s
+    And pktvisor API must be enabled                   # features/steps/pktvisor.py:75 1.123s
+
+  Scenario: run multiple pktvisors using different ports  # features/pktvisor.feature:8
+    When run pktvisor instance on port default            # features/steps/pktvisor.py:33 0.156s
+    And run pktvisor instance on port 10854               # features/steps/pktvisor.py:33 0.127s
+    And run pktvisor instance on port 10855               # features/steps/pktvisor.py:33 0.146s
+    Then all the pktvisor containers must be running      # features/steps/pktvisor.py:47 0.011s
+    And 3 pktvisor's containers must be running           # features/steps/pktvisor.py:59 0.012s
+
+  Scenario: run multiple pktvisors instances using the same port  # features/pktvisor.feature:15
+    When run pktvisor instance on port default                    # features/steps/pktvisor.py:33 0.194s
+    And run pktvisor instance on port default                     # features/steps/pktvisor.py:33 0.149s
+    Then 1 pktvisor's containers must be running                  # features/steps/pktvisor.py:59 0.226s
+    And 1 pktvisor's containers must be exited                    # features/steps/pktvisor.py:59 0.011s
+
+1 feature passed, 0 failed, 0 skipped
+3 scenarios passed, 0 failed, 0 skipped
+12 steps passed, 0 failed, 0 skipped, 0 undefined
+Took 0m2.312s
+
+
+```

automated_tests/docs/development_guide.md

@@ -0,0 +1,9 @@
+
+## **PKTVISOR**
+
+
+|                                                      Scenario                                                      | Automated | Smoke | Sanity | 
+|:------------------------------------------------------------------------------------------------------------------:|:---------:|:-----:|:------:|
+|            [Run pktvisor instance using docker command](pktvisor/run_pktvisor_instance_using_docker_command.md)             |    👍     |  👍   |   👍   |   
+| [Run multiple pktvisors instances using different ports](pktvisor/run_multiple_pktvisors_instances_using_different_ports.md) |    👍     |  👍   |   👍   |
+|  [Run multiple pktvisors instances using the same ports](pktvisor/run_multiple_pktvisors_instances_using_the_same_ports.md)  |    👍     |  👍   |   👍   |

automated_tests/docs/pktvisor/run_multiple_pktvisors_instances_using_different_ports.md

@@ -0,0 +1,11 @@
+## Scenario: Run multiple pktvisors instances using different ports 
+
+## Steps:
+- Provide 1 pktvisor instance using `docker run --net=host -d ns1labs/pktvisor pktvisord <net>`
+- Provide 1 pktvisor instance using `docker run --net=host -d ns1labs/pktvisor pktvisord -p 10854 <net>`
+
+
+## Expected Result:
+- Both pktvisor containers must be running (one on port 10853 and one on port 10854)
+- Endpoints from pktvisor API must be accessible (port 10853 and 10854)
+

automated_tests/docs/pktvisor/run_multiple_pktvisors_instances_using_the_same_ports.md

@@ -0,0 +1,11 @@
+## Scenario: Run multiple pktvisors instances using the same ports 
+
+## Steps:
+- Provide 1 pktvisor instance using `docker run --net=host -d ns1labs/pktvisor pktvisord <net>`
+- Provide 1 pktvisor instance using `docker run --net=host -d ns1labs/pktvisor pktvisord <net>`
+
+
+## Expected Result:
+- The first pktvisor instance provisioned must be running (one on port 10853)
+- Second pktvisor container must be exited
+

automated_tests/docs/pktvisor/run_pktvisor_instance_using_docker_command.md

@@ -0,0 +1,9 @@
+## Scenario: Run pktvisor instance using docker command 
+## Steps: 
+ - Run docker using `docker run --net=host -d ns1labs/pktvisor pktvisord <net>`
+
+
+## Expected Result:
+ - Pktvisor container must be running
+ - Endpoints from pktvisor API must be accessible
+

automated_tests/features/behave.ini

@@ -0,0 +1,4 @@
+[behave]
+stderr_capture=False
+stdout_capture=False
+

automated_tests/features/environment.py

@@ -0,0 +1,22 @@
+import docker
+
+
+PKTVISOR_CONTAINER_NAME = "pktvisor-test"
+
+
+def before_scenario(context, scenario):
+    cleanup_container(PKTVISOR_CONTAINER_NAME)
+
+
+def after_scenario(context, feature):
+    cleanup_container(PKTVISOR_CONTAINER_NAME)
+
+
+def cleanup_container(name_prefix):
+    docker_client = docker.from_env()
+    containers = docker_client.containers.list(all=True)
+    for container in containers:
+        test_container = container.name.startswith(name_prefix)
+        if test_container is True:
+            container.stop()
+            container.remove()

automated_tests/features/pktvisor.feature

@@ -0,0 +1,19 @@
+Feature: pktvisor tests
+
+Scenario: pktvisor bootstrap
+  When run pktvisor instance on port default
+  Then the pktvisor container status must be running
+    And pktvisor API must be enabled
+
+Scenario: run multiple pktvisors using different ports
+  When run pktvisor instance on port default
+    And run pktvisor instance on port 10854
+    And run pktvisor instance on port 10855
+  Then all the pktvisor containers must be running
+    And 3 pktvisor's containers must be running
+
+Scenario: run multiple pktvisors instances using the same port
+  When run pktvisor instance on port default
+    And run pktvisor instance on port default
+  Then 1 pktvisor's containers must be running
+    And 1 pktvisor's containers must be exited

automated_tests/features/steps/pktvisor.py

@@ -0,0 +1,104 @@
+from utils import random_string
+import docker
+from behave import step
+from hamcrest import *
+import requests
+from retry import retry
+
+PKTVISOR_CONTAINER_NAME = "pktvisor-test"
+
+
+def run_pktvisor_container(container_image, port="default", container_name=PKTVISOR_CONTAINER_NAME):
+    """
+    Run a pktvisor container
+
+    :param (str) container_image: that will be used for running the container
+    :param (str) port: Port on which the web service must be run [default: 10853]
+    :param (dict) env_vars: that will be passed to the container context
+    :param (str) container_name: base of container name
+    :returns: (str) the container ID
+    """
+    PKTVISOR_CONTAINER_NAME = container_name + random_string(3)
+    client = docker.from_env()
+    pkt_command = ["pktvisord", "wlo1"]
+    if port != "default":
+        pkt_command.insert(-1, '-p')
+        pkt_command.insert(-1, port)
+    container = client.containers.run(container_image, name=PKTVISOR_CONTAINER_NAME, detach=True,
+                                      network_mode='host', command=pkt_command)
+    return container.id
+
+
+
+@step("run pktvisor instance on port {pkt_port}")
+def run_pktvisor(context, pkt_port):
+    context.pkt_port = pkt_port
+    context.container_id = run_pktvisor_container("ns1labs/pktvisor", pkt_port)
+
+
+@step("the pktvisor container status must be {pkt_status}")
+def check_pkt_status(context, pkt_status):
+    docker_client = docker.from_env()
+    container = docker_client.containers.get(context.container_id)
+    status = container.status
+    assert_that(status, equal_to(pkt_status), f"pktvisor container {context.container_id} failed with status:{status}")
+
+
+@step("all the pktvisor containers must be {pkt_status}")
+def check_pkt_status(context, pkt_status):
+    docker_client = docker.from_env()
+
+    containers = docker_client.containers.list(all=True)
+    for container in containers:
+        is_test_container = container.name.startswith(PKTVISOR_CONTAINER_NAME)
+        if is_test_container is True:
+            status = container.status
+            assert_that(status, equal_to(pkt_status), f"pktvisor container {container.id} failed with status:{status}")
+
+
+@step("{amount_of_pktvisor} pktvisor's containers must be {pkt_status}")
+@retry(tries=5, delay=0.2)
+def check_amount_of_pkt_with_status(context, amount_of_pktvisor, pkt_status):
+    docker_client = docker.from_env()
+    containers = docker_client.containers.list(all=True)
+    containers_with_expected_status = list()
+    for container in containers:
+        is_test_container = container.name.startswith(PKTVISOR_CONTAINER_NAME)
+        if is_test_container is True:
+            status = container.status
+            if status == pkt_status:
+                containers_with_expected_status.append(container)
+    assert_that(len(set(containers_with_expected_status)), equal_to(int(amount_of_pktvisor)),
+                f"Amount of pktvisor container with referred status failed")
+
+
+@step("pktvisor API must be enabled")
+def check_pkt_base_API(context):
+    if context.pkt_port == "default":
+        context.pkt_port = 10853
+
+    pkt_api_get_endpoints = ['metrics/app',
+                             'metrics/bucket/0',
+                             'metrics/window/2',
+                             'metrics/window/3',
+                             'metrics/window/4',
+                             'metrics/window/5',
+                             'taps',
+                             'policies',
+                             'policies/__all/metrics/window/2',
+                             'policies/__all/metrics/window/3',
+                             'policies/__all/metrics/window/4',
+                             'policies/__all/metrics/window/5',
+                             'policies/__all/metrics/prometheus']
+    for endpoint in pkt_api_get_endpoints:
+        make_get_request(endpoint, context.pkt_port)
+
+
+@retry(tries=3, delay=1)
+def make_get_request(end_point, pkt_port=10853, expected_status_code=200):
+    pkt_base_api = 'http://localhost:'+str(pkt_port)+'/api/v1/'
+    path = pkt_base_api+end_point
+    response = requests.get(path)
+    assert_that(response.status_code, equal_to(int(expected_status_code)),
+                f"Get request to endpoint {path} failed with status {response.status_code}")
+    return response

automated_tests/features/steps/utils.py

@@ -0,0 +1,60 @@
+import random
+import string
+from json import loads, JSONDecodeError
+
+
+def random_string(k=10):
+    """
+    Generates a string composed of of k (int) random letters lowercase and uppercase mixed
+
+    :param (int) k: sets the length of the randomly generated string
+    :return: (str) string consisting of k random letters lowercase and uppercase mixed. Default:10
+    """
+    return ''.join(random.choices(string.ascii_letters, k=k))
+
+
+def safe_load_json(json_str):
+    """
+    Safely parses a string into a JSON object, without ever raising an error.
+    :param (str) json_str: to be loaded
+    :return: the JSON object, or None if string is not a valid JSON.
+    """
+
+    try:
+        return loads(json_str)
+    except JSONDecodeError:
+        return None
+
+
+def check_logs_contain_message_and_name(logs, expected_message, name, name_key):
+    """
+    Gets the logs from Orb agent container
+
+    :param (list) logs: list of log lines
+    :param (str) expected_message: message that we expect to find in the logs
+    :param (str) name: element name that we expect to find in the logs
+    :param (str) name_key: key to get element name on log line
+    :returns: (bool) whether expected message was found in the logs
+    """
+
+    for log_line in logs:
+        log_line = safe_load_json(log_line)
+
+        if log_line is not None and log_line['msg'] == expected_message:
+            if log_line is not None and log_line[name_key] == name:
+                return True, log_line
+
+    return False, "Logs doesn't contain the message and name expected"
+
+
+def remove_empty_from_json(json_file):
+    """
+    Delete keys with the value "None" in a dictionary, recursively.
+
+    """
+    for key, value in list(json_file.items()):
+        if value is None:
+            del json_file[key]
+        elif isinstance(value, dict):
+            remove_empty_from_json(value)
+    return json_file

automated_tests/requirements.txt

@@ -0,0 +1,15 @@
+behave==1.2.6
+certifi==2021.10.8
+charset-normalizer==2.0.12
+decorator==5.1.1
+docker==5.0.3
+idna==3.3
+parse==1.19.0
+parse-type==0.6.0
+py==1.11.0
+PyHamcrest==2.0.3
+requests==2.27.1
+retry==0.9.2
+six==1.16.0
+urllib3==1.26.9
+websocket-client==1.3.1