small code re-org and updated docs for directory walking (#9)

* small code re-org and updated docs for directory walking --------- Co-authored-by: Matthew Warren <matt.warren+github-commits@cloudtruth.com>
cloudtruth · Mar 11, 2024 · 61aaf6f · 61aaf6f
1 parent 818ab71
commit 61aaf6f
Show file tree

Hide file tree

Showing 15 changed files with 150 additions and 118 deletions.
diff --git a/README.md b/README.md
@@ -24,28 +24,35 @@ This utility is distributed as a Docker container and can be pulled from cloudtr
 ## Processing a single file
 An example of how to process a .env file
 ```
-docker run --rm -v ${PWD}/files:/app/files cloudtruth/dynamic-importer process-configs --default-values /app/dynamic_importer/samples/.env.sample --file-type dotenv --output-dir /app/files/
+docker run --rm -v ${PWD}/files:/app/files cloudtruth/dynamic-importer process-configs -p myproj --default-values /app/samples/.env.sample --file-type dotenv --output-dir /app/files/
 ```
 
 This command will mount a subdir `files` from the current working directory to the container. Assuming your input file is in that dir, the processed files will be placed in that dir once processing has completed.
 
 ## Processing several files
 An example of how to orocess several .env files and create values for each environment
 ```
-docker run --rm -v ${PWD}/files:/app/files cloudtruth/dynamic-importer process-configs -t dotenv \
-    --default-values /app/dynamic_importer/samples/dotenvs/.env.default.sample \
-    --env-values development:/app/dynamic_importer/samples/dotenvs/.env.dev.sample \
-    --env-values staging:/app/dynamic_importer/samples/dotenvs/.env.staging.sample \
-    --env-values production:/app/dynamic_importer/samples/dotenvs/.env.prod.sample \
+docker run --rm -v ${PWD}/files:/app/files cloudtruth/dynamic-importer process-configs \
+    -p myproj -t dotenv \
+    --default-values /app/samples/dotenvs/.env.default.sample \
+    --env-values development:/app/samples/dotenvs/.env.dev.sample \
+    --env-values staging:/app/samples/dotenvs/.env.staging.sample \
+    --env-values production:/app/samples/dotenvs/.env.prod.sample \
     --output-dir /app/files/
 ```
 
+## Procesing a directory tree
+You may also feed a directory of files into the `walk-directories` command, which will find all files matching the supplied types and parse them into CloudTruth config formats.
+
+```
+docker run --rm -v ${PWD}/files:/app/files cloudtruth/dynamic-importer walk-directories -c /app/samples/ -t dotenv -t json -t tf -o walk_output/
+```
 
 ## Editing template references
 There may be times when this utility is too aggressive or you want a variable to remain hard-coded in your CloudTruth template. In that case, you can remove the references from the generated `.ctconfig` file and re-generate the template.
 
 ```
-docker run --rm -v ${PWD}/files:/app/files cloudtruth/dynamic-importer regenerate-template --input-file /app/dynamic_importer/samples/.env.sample --file-type dotenv --data-file /app/files/.env.ctconfig --output-dir /app/files/
+docker run --rm -v ${PWD}/files:/app/files cloudtruth/dynamic-importer regenerate-template --input-file /app/samples/.env.sample --file-type dotenv --data-file /app/files/.env.ctconfig --output-dir /app/files/
 ```
 
 ## Uploading data to CloudTruth
@@ -64,12 +71,14 @@ By default, the utility will prompt for your CloudTruth API Key. You may also pr
 # Testing
 Test code lives in `src/tests` and uses [click.testing](https://click.palletsprojects.com/en/8.1.x/testing/) as the entrypoint for all commands and processors. There are additional unit tests for the api client code, which heavily leverages mocks for the CloudTruth API. See examples in `tests.fixures.requests` for more.
 
+To run unittests, run `pytest` from within your virtualenv.
+
 Pre-commit is installed in this repo and should be used to verify code organization and formatting. To set it up, run `pre-commit install` in your virtualenv
 
 # Contributing
-Issues, pull requests, and discussions are welcomed.
+Issues, pull requests, and discussions are welcomed. Please vote for any issues tagged with [needs votes](https://github.com/cloudtruth/dynamic-importer/issues?q=is%3Aissue+is%3Aopen+label%3A%22needs+votes%22)
 
-See dynamic_importer.processors and the subclasses within for examples of the current design. TL;DR - if you can convert the source into a dict, BaseProcessor._traverse_data should handle most of the heavy lifting.
+See `dynamic_importer.processors` and the subclasses within for examples of the current design. TL;DR - if you can convert the source into a dict, `BaseProcessor._traverse_data` should handle most of the heavy lifting.
 
 # Counter-examples
 Because this has come up internally, this utility is intended to process config data, not application code or even raw IaC code. If you want to feed it a full Terraform manifest, you're going to get strange results. Pull your common variables into a variables.tf first!
diff --git a/pyproject.toml b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "cloudtruth-dynamic-importer"
-version = "0.3.0"
+version = "0.5.0"
 dependencies = [
     "click",
     "python-dotenv @ git+https://github.com/cloudtruth/python-dotenv@feature/dump-dotenv",
@@ -22,6 +22,7 @@ dev = [
     "pytest-black",
     "pytest-cov",
     "pytest-mypy",
+    "pytest-timeout",
     "types-requests",
     "types-pyyaml",
 ]

diff --git a/src/samples/.env.sample → samples/.env.sample b/src/samples/.env.sample → samples/.env.sample
diff --git a/src/samples/azureTRE.yaml → samples/azureTRE.yaml b/src/samples/azureTRE.yaml → samples/azureTRE.yaml
diff --git a/src/samples/dotenvs/.env.default.sample → samples/dotenvs/.env.default.sample b/src/samples/dotenvs/.env.default.sample → samples/dotenvs/.env.default.sample
diff --git a/src/samples/dotenvs/.env.dev.sample → samples/dotenvs/.env.dev.sample b/src/samples/dotenvs/.env.dev.sample → samples/dotenvs/.env.dev.sample
diff --git a/src/samples/dotenvs/.env.prod.sample → samples/dotenvs/.env.prod.sample b/src/samples/dotenvs/.env.prod.sample → samples/dotenvs/.env.prod.sample
diff --git a/src/samples/dotenvs/.env.staging.sample → samples/dotenvs/.env.staging.sample b/src/samples/dotenvs/.env.staging.sample → samples/dotenvs/.env.staging.sample
diff --git a/src/samples/short.json → samples/short.json b/src/samples/short.json → samples/short.json
diff --git a/src/samples/terraform.tfvars → samples/terraform.tfvars b/src/samples/terraform.tfvars → samples/terraform.tfvars
diff --git a/src/samples/variables.tf → samples/variables.tf b/src/samples/variables.tf → samples/variables.tf
diff --git a/src/dynamic_importer/main.py b/src/dynamic_importer/main.py
@@ -12,6 +12,7 @@
 from dynamic_importer.processors import get_processor_class
 from dynamic_importer.processors import get_supported_formats
 from dynamic_importer.util import validate_env_values
+from dynamic_importer.walker import walk_files
 
 CREATE_DATA_MSG_INTERVAL = 10
 DIRS_TO_IGNORE = [
@@ -25,15 +26,6 @@
     "build",
     "target",
 ]
-# mime types think .env and tf files are plain text
-EXTENSIONS_TO_FILE_TYPES = {
-    ".json": "json",
-    ".yaml": "yaml",
-    ".yml": "yaml",
-    ".env": "dotenv",
-    ".tf": "tf",
-    ".tfvars": "tfvars",
-}
 
 
 @click.group()
@@ -70,7 +62,10 @@ def import_config():
     default=".",
     required=False,
 )
-def process_configs(file_type, default_values, env_values, output_dir):
+@click.option(
+    "-p", "--project", help="CloudTruth project to import data into", required=True
+)
+def process_configs(file_type, default_values, env_values, output_dir, project):
     if not default_values and not env_values:
         raise click.UsageError(
             "At least one of --default-values and --env-values must be provided"
@@ -88,11 +83,8 @@ def process_configs(file_type, default_values, env_values, output_dir):
     processor: BaseProcessor = processing_class(input_files)
     template, config_data = processor.process()
 
-    input_filename = ".".join(
-        list(input_files.values())[0].split("/")[-1].split(".")[:-1]
-    )
-    template_out_file = f"{output_dir}/{input_filename}.cttemplate"
-    config_out_file = f"{output_dir}/{input_filename}.ctconfig"
+    template_out_file = f"{output_dir}/{project}-{file_type}.cttemplate"
+    config_out_file = f"{output_dir}/{project}-{file_type}.ctconfig"
 
     click.echo(f"Writing template to: {template_out_file}")
     with open(template_out_file, "w+") as fp:
@@ -253,49 +245,13 @@ def walk_directories(config_dir, file_types, output_dir):
     output_dir = output_dir.rstrip("/")
     for root, dirs, files in os.walk(config_dir):
         root = root.rstrip("/")
-        last_project = None
 
         # skip over known non-config directories
         for dir in DIRS_TO_IGNORE:
             if dir in dirs:
                 dirs.remove(dir)
 
-        for file in files:
-            file_path = f"{root}/{file}"
-            name, file_extension = os.path.splitext(file)
-            if name.startswith(".env"):
-                file_extension = ".env"
-            if data_type := EXTENSIONS_TO_FILE_TYPES.get(file_extension):
-                confirmed_type = click.prompt(
-                    f"File type {data_type} detected for {file_path}. Is this correct?",
-                    type=click.Choice(get_supported_formats(), case_sensitive=False),
-                    default=data_type,
-                )
-                if confirmed_type not in file_types:
-                    click.echo(
-                        f"Skipping {confirmed_type} file {file_path} as "
-                        f"it is not included in the supplied file types: {', '.join(file_types)}"
-                    )
-                    continue
-                project = click.prompt(
-                    f"Enter the CloudTruth project to import {file_path} into",
-                    default=last_project,
-                )
-                if project:
-                    last_project = project
-                env = click.prompt(
-                    f"Enter the CloudTruth environment to import {file_path} into",
-                )
-                if not env:
-                    env = click.prompt(
-                        "Environment cannot be empty. Please enter a CloudTruth environment"
-                    )
-                walked_files[file_path] = {
-                    "type": confirmed_type,
-                    "path": file_path,
-                    "project": project,
-                    "environment": env,
-                }
+        walked_files.update(walk_files(root, files, file_types))
 
     project_files = defaultdict(lambda: defaultdict(list))
     for v in walked_files.values():
@@ -305,15 +261,15 @@ def walk_directories(config_dir, file_types, output_dir):
     for project, type_info in project_files.items():
         for file_type, file_meta in type_info.items():
             env_paths = {d["environment"]: d["path"] for d in file_meta}
-            input_filename = ".".join(
-                list(env_paths.values())[0].split("/")[-1].split(".")[:-1]
-            )
+
             click.echo(f"Processing {project} files: {', '.join(env_paths.values())}")
             processing_class = get_processor_class(file_type)
             processor: BaseProcessor = processing_class(env_paths)
             template, config_data = processor.process()
-            template_out_file = f"{output_dir}/{input_filename}.cttemplate"
-            config_out_file = f"{output_dir}/{input_filename}.ctconfig"
+
+            template_out_file = f"{output_dir}/{project}-{file_type}.cttemplate"
+            config_out_file = f"{output_dir}/{project}-{file_type}.ctconfig"
+
             click.echo(f"Writing template to: {template_out_file}")
             with open(template_out_file, "w+") as fp:
                 template_body = processor.generate_template()

diff --git a/src/dynamic_importer/walker.py b/src/dynamic_importer/walker.py
@@ -0,0 +1,60 @@
+from __future__ import annotations
+
+import os
+from typing import Dict
+from typing import List
+
+import click
+from dynamic_importer.processors import get_supported_formats
+
+
+# mime types think .env and tf files are plain text
+EXTENSIONS_TO_FILE_TYPES = {
+    ".json": "json",
+    ".yaml": "yaml",
+    ".yml": "yaml",
+    ".env": "dotenv",
+    ".tf": "tf",
+    ".tfvars": "tfvars",
+}
+
+
+def walk_files(
+    root: str, files: str, file_types: List[str]
+) -> Dict[str, Dict[str, str]]:
+    walked_files = {}
+    last_project = None
+    for file in files:
+        file_path = f"{root}/{file}"
+        name, file_extension = os.path.splitext(file)
+        if name.startswith(".env"):
+            file_extension = ".env"
+        if data_type := EXTENSIONS_TO_FILE_TYPES.get(file_extension):
+            confirmed_type = click.prompt(
+                f"File type {data_type} detected for {file_path}. Is this correct?",
+                type=click.Choice(get_supported_formats(), case_sensitive=False),
+                default=data_type,
+            )
+            if confirmed_type not in file_types:
+                click.echo(
+                    f"Skipping {confirmed_type} file {file_path} as "
+                    f"it is not included in the supplied file types: {', '.join(file_types)}"
+                )
+                continue
+            project = click.prompt(
+                f"Enter the CloudTruth project to import {file_path} into",
+                default=last_project,
+            )
+            if project:
+                last_project = project
+            env = click.prompt(
+                f"Enter the CloudTruth environment to import {file_path} into",
+            )
+            walked_files[file_path] = {
+                "type": confirmed_type,
+                "path": file_path,
+                "project": project,
+                "environment": env,
+            }
+
+    return walked_files
diff --git a/src/tests/test_cli.py b/src/tests/test_cli.py
@@ -1,7 +1,6 @@
 from __future__ import annotations
 
 import pathlib
-import traceback
 from unittest import mock
 from unittest import TestCase
 
@@ -28,7 +27,9 @@ def test_cli_help(self):
 
     def test_process_configs_no_args(self):
         runner = CliRunner()
-        result = runner.invoke(import_config, ["process-configs", "-t", "dotenv"])
+        result = runner.invoke(
+            import_config, ["process-configs", "-t", "dotenv", "-p", "testproj"]
+        )
         self.assertEqual(result.exit_code, 2)
         self.assertIn(
             "Error: At least one of --default-values and --env-values must be provided",
@@ -58,8 +59,10 @@ def test_cli_process_configs_dotenv(tmp_path):
                 "process-configs",
                 "-t",
                 "dotenv",
+                "-p",
+                "testproj",
                 "--default-values",
-                f"{current_dir}/../samples/.env.sample",
+                f"{current_dir}/../../samples/.env.sample",
                 "--output-dir",
                 td,
             ],
@@ -79,8 +82,10 @@ def test_cli_process_configs_json(tmp_path):
                 "process-configs",
                 "-t",
                 "json",
+                "-p",
+                "testproj",
                 "--default-values",
-                f"{current_dir}/../samples/short.json",
+                f"{current_dir}/../../samples/short.json",
                 "--output-dir",
                 td,
             ],
@@ -100,8 +105,10 @@ def test_cli_process_configs_tf(tmp_path):
                 "process-configs",
                 "-t",
                 "tf",
+                "-p",
+                "testproj",
                 "--default-values",
-                f"{current_dir}/../samples/variables.tf",
+                f"{current_dir}/../../samples/variables.tf",
                 "--output-dir",
                 td,
             ],
@@ -121,8 +128,10 @@ def test_cli_process_configs_tfvars(tmp_path):
                 "process-configs",
                 "-t",
                 "tfvars",
+                "-p",
+                "testproj",
                 "--default-values",
-                f"{current_dir}/../samples/terraform.tfvars",
+                f"{current_dir}/../../samples/terraform.tfvars",
                 "--output-dir",
                 td,
             ],
@@ -142,8 +151,10 @@ def test_cli_process_configs_yaml(tmp_path):
                 "process-configs",
                 "-t",
                 "yaml",
+                "-p",
+                "testproj",
                 "--default-values",
-                f"{current_dir}/../samples/azureTRE.yaml",
+                f"{current_dir}/../../samples/azureTRE.yaml",
                 "--output-dir",
                 td,
             ],
@@ -158,9 +169,9 @@ def test_cli_process_configs_yaml(tmp_path):
                 "-t",
                 "yaml",
                 "--default-values",
-                f"{current_dir}/../samples/azureTRE.yaml",
+                f"{current_dir}/../../samples/azureTRE.yaml",
                 "--data-file",
-                f"{td}/azureTRE.ctconfig",
+                f"{td}/testproj-yaml.ctconfig",
             ],
             catch_exceptions=False,
         )
@@ -188,8 +199,10 @@ def test_cli_import_data_json(mock_get, mock_post, tmp_path):
                 "process-configs",
                 "-t",
                 "json",
+                "-p",
+                "testproj",
                 "--default-values",
-                f"{current_dir}/../samples/short.json",
+                f"{current_dir}/../../samples/short.json",
                 "-o",
                 td,
             ],
@@ -202,17 +215,12 @@ def test_cli_import_data_json(mock_get, mock_post, tmp_path):
             [
                 "create-data",
                 "-d",
-                f"{td}/short.ctconfig",
+                f"{td}/testproj-json.ctconfig",
                 "-m",
-                f"{td}/short.cttemplate",
+                f"{td}/testproj-json.cttemplate",
                 "-p",
                 "testproj",
             ],
             catch_exceptions=False,
         )
-        try:
-            assert result.exit_code == 0
-        except AssertionError:
-            print(result.output)
-            print(traceback.format_tb(result.exc_info[2]))
-            raise
+        assert result.exit_code == 0