Implementing application level configuration parsing + deployment seq… #413

PhillypHenning · 2023-03-02T16:59:35Z

…uence

Description

Users should be able to specify an alternative deployment sequence within their application repo.

Fixes # (issue)
fixes #410
closes #367
closes #259

Type of change

New feature (non-breaking change which adds functionality)
Documentation update ???

How Has This Been Tested?

Logs
Logs are posted in the linked ticket.

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation

…uence

runforesight · 2023-03-02T17:01:37Z

Foresight Summary

Major Impacts
Foresight hasn't detected any major impact on your workflows and tests.

View More Details

✅ CI workflow has finished in 35 seconds and finished at 8th Mar, 2023.

Job	Failed Steps	Tests
lint-pylint	- 🔗	N/A	See Details
tests-unit	- 🔗	N/A	See Details
lint-black	- 🔗	N/A	See Details

🔎 See details on Foresight

^{*You can configure Foresight comments in your organization settings page.}

arm4b · 2023-03-02T18:01:51Z

scripts/plugins/deploy_plugins.py

+    # Check if application level bitops.config exists
+    application_bitops_configuration_path = f"{bitops_operations_dir}/bitops.config.yaml"
+    application_configuration = None
+    if os.path.exists(application_bitops_configuration_path):
+        with open(application_bitops_configuration_path, "r", encoding="utf8") as stream:
+            application_config_yaml = yaml.load(stream, Loader=yaml.FullLoader)
+        application_configuration = DefaultMunch.fromDict(application_config_yaml, None)


This will work.
It will work only for deployments section though.

If we're allowing users to include their bitops.config.yaml on a root ops repo level, I'd assume full config should really work.

Is there a place where bitops.config.yaml is parsed at earlier time? We could try to load the user's config it there.

We'll benefit from it when users want to also set the logging level, secrets masking or similar on BitOps runtime.

This loads the user level configuration.
AND THEN it populates the potential deployment sequence.

This was made with the above further enhancement in mind in that, we not have a loaded user application repo level configuration.

I did not go further then that, however the work is in a place where further enhancements to user specified values wouldn't be challenging

I'd prefer to create a secondary ticket for that feature, what are your thoughts Eugen

I'm talking about this part where the Docker-packaged bitops.config.yaml is loaded at early stage.

https://github.com/bitovi/bitops/blob/main/scripts/plugins/settings.py#L14-L35

If we try to load user-defined bitops.config.yaml config from opsrepo right after that point, - it'll work for the majority of bitops.config settings, not just deployments

The bitops configuration loads in the settings.py

bitops/scripts/plugins/settings.py

Line 34 in a8e834b

bitops_build_configuration = DefaultMunch.fromDict(BITOPS_config_yaml, None)

It's not possible to have it in the settings.py

I'll look into doing the application bitops configuration load earlier (to gain access to the user specified variables).

@armab looking at the code, I'm not sure we could raise the application level loading any higher.
Additionally (unless I missed something) all potentially used variables that would be modifiable from the application level configuration would be done after the current block.

So in short, what you're asking for already takes place in a position that we could further enhance BitOps configuration before the "deployment" begins.

The ENV variables to override the bitops.config.yaml from settings are these:

bitops/scripts/plugins/settings.py

Lines 58 to 123 in a8e834b

BITOPS_fast_fail_mode = (

BITOPS_ENV_fast_fail_mode

if BITOPS_ENV_fast_fail_mode is not None

else bitops_build_configuration.bitops.fail_fast

if bitops_build_configuration.bitops.fail_fast is not None

else True

)

BITOPS_run_mode = (

BITOPS_ENV_run_mode

if BITOPS_ENV_run_mode is not None

else bitops_build_configuration.bitops.run_mode

if bitops_build_configuration.bitops.run_mode is not None

else "default"

)

BITOPS_logging_level = (

BITOPS_ENV_logging_level

if BITOPS_ENV_logging_level is not None

else bitops_build_configuration.bitops.logging.level

if bitops_build_configuration.bitops.logging.level is not None

else "DEBUG"

)

BITOPS_logging_color = (

bitops_build_configuration.bitops.logging.color.enabled

if bitops_build_configuration.bitops.logging.color.enabled is not None

else False

)

BITOPS_logging_filename = (

bitops_build_configuration.bitops.logging.filename

if bitops_build_configuration.bitops.logging.filename is not None

else None

)

BITOPS_logging_path = (

bitops_build_configuration.bitops.logging.path

if bitops_build_configuration.bitops.logging.path is not None

else "/var/log/bitops"

)

BITOPS_plugin_dir = (

BITOPS_ENV_plugin_dir

if BITOPS_ENV_plugin_dir is not None

else bitops_build_configuration.bitops.plugins.plugin_dir

if bitops_build_configuration.bitops.plugins.plugin_dir is not None

else "/opt/bitops/scripts/plugins/"

)

BITOPS_INSTALLED_PLUGINS_DIR = "/opt/bitops/scripts/installed_plugins/"

BITOPS_default_folder = (

BITOPS_ENV_default_folder

if BITOPS_ENV_default_folder is not None

else bitops_build_configuration.bitops.default_folder

if bitops_build_configuration.bitops.default_folder is not None

else "_default"

)

BITOPS_timeout = (

BITOPS_ENV_timeout

if BITOPS_ENV_timeout is not None

else bitops_build_configuration.bitops.timeout

if bitops_build_configuration.bitops.timeout is not None

else 600

This way the precedence would be the following:

Docker-bundled bitops.config.yaml -> User-specified OpsRepo bitops.config.yaml (new) -> ENV variables

Where ENV variables have priority.

It's not possible to make this kind of expected full config vs ENV variable precedence before moving the user-provided configuration at a earlier level.

I also looked into the code, and it should be doable to read the user-provided ops-repo bitops.config.yaml at that level.

It's fine if we do it before /tmp creation, as we're just reading the config and that's all (no writes involved).

arm4b · 2023-03-08T15:58:34Z

scripts/plugins/settings.py

+def parse_config(dictionary, dotted_key_list, validate=False):
+    """
+    This function takes a dictionary, a list of keys in dotted notation,
+    and an optional boolean argument "validate". It uses the operator.attrgetter()
+    method to access the value in the dictionary associated with the dotted key list.
+    """
+    try:
+        item = operator.attrgetter(dotted_key_list)(dictionary)
+        if item is None and validate:
+            return False
+        if item is not None and validate:
+            return True
+        return item
+    except AttributeError:
+        # Likely cause: Nested value doesn't exist
+        if validate:
+            return False
+        return None


As I see validate is just a None check.

Do we really need it?

For example:

parse_config(bitops_build_configuration, "bitops.fail_fast") if parse_config(bitops_build_configuration, "bitops.fail_fast", validate=True)

is the same as

parse_config(bitops_build_configuration, "bitops.fail_fast") if parse_config(bitops_build_configuration, "bitops.fail_fast") is not None

right?

Good point 👍

…ue from the chain of vars

…thub.com/bitovi/bitops into 410-application-repo-deployment-sequence

arm4b

Thanks for the changes!

Implementing application level configuration parsing + deployment seq…

15439ca

…uence

PhillypHenning linked an issue Mar 2, 2023 that may be closed by this pull request

Application repo Deployment sequence #410

Closed

PhillypHenning requested review from arm4b and LeoDiazL March 2, 2023 16:59

PhillypHenning added the enhancement ✨ New feature or request label Mar 2, 2023

PhillypHenning added this to the v2.5.0 milestone Mar 2, 2023

arm4b reviewed Mar 2, 2023

View reviewed changes

arm4b force-pushed the 410-application-repo-deployment-sequence branch from 15439ca to 1490c7a Compare March 7, 2023 19:07

Final touches on user specified bitops configuration

4f10bbd

PhillypHenning force-pushed the 410-application-repo-deployment-sequence branch from 1490c7a to 4f10bbd Compare March 7, 2023 23:20

PhillypHenning added 2 commits March 7, 2023 18:50

lint+formatting

e27beb8

Capitializing all constants

e19af39

arm4b reviewed Mar 8, 2023

View reviewed changes

PhillypHenning and others added 6 commits March 8, 2023 11:31

Removing validation check (rely on None value)

ee1bb05

Linting+formatting cleanup

92e579c

Add get_first() function to simplify getting the first meaningful val…

b5e4316

…ue from the chain of vars

Merge branch '410-application-repo-deployment-sequence' of https://gi…

2df50a2

…thub.com/bitovi/bitops into 410-application-repo-deployment-sequence

Fix black formatting

955466f

Adding OpsRepo config mention to configuration doc

f41d4f3

PhillypHenning force-pushed the 410-application-repo-deployment-sequence branch from 43b777b to f41d4f3 Compare March 8, 2023 17:19

Add a note about deployment sequence control to Release History

c6c341f

arm4b approved these changes Mar 8, 2023

View reviewed changes

PhillypHenning merged commit 2cb5a78 into main Mar 8, 2023

PhillypHenning deleted the 410-application-repo-deployment-sequence branch March 8, 2023 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementing application level configuration parsing + deployment seq… #413

Implementing application level configuration parsing + deployment seq… #413

PhillypHenning commented Mar 2, 2023 •

edited by arm4b

Loading

runforesight bot commented Mar 2, 2023 •

edited

Loading

✅ CI workflow has finished in 35 seconds and finished at 8th Mar, 2023.

arm4b Mar 2, 2023

PhillypHenning Mar 2, 2023

PhillypHenning Mar 2, 2023

arm4b Mar 2, 2023 •

edited

Loading

PhillypHenning Mar 2, 2023

PhillypHenning Mar 2, 2023

arm4b Mar 7, 2023 •

edited

Loading

arm4b Mar 7, 2023

arm4b Mar 8, 2023

PhillypHenning Mar 8, 2023

arm4b left a comment

	BITOPS_fast_fail_mode = (
	BITOPS_ENV_fast_fail_mode
	if BITOPS_ENV_fast_fail_mode is not None
	else bitops_build_configuration.bitops.fail_fast
	if bitops_build_configuration.bitops.fail_fast is not None
	else True
	)

	BITOPS_run_mode = (
	BITOPS_ENV_run_mode
	if BITOPS_ENV_run_mode is not None
	else bitops_build_configuration.bitops.run_mode
	if bitops_build_configuration.bitops.run_mode is not None
	else "default"
	)

	BITOPS_logging_level = (
	BITOPS_ENV_logging_level
	if BITOPS_ENV_logging_level is not None
	else bitops_build_configuration.bitops.logging.level
	if bitops_build_configuration.bitops.logging.level is not None
	else "DEBUG"
	)

	BITOPS_logging_color = (
	bitops_build_configuration.bitops.logging.color.enabled
	if bitops_build_configuration.bitops.logging.color.enabled is not None
	else False
	)

	BITOPS_logging_filename = (
	bitops_build_configuration.bitops.logging.filename
	if bitops_build_configuration.bitops.logging.filename is not None
	else None
	)

	BITOPS_logging_path = (
	bitops_build_configuration.bitops.logging.path
	if bitops_build_configuration.bitops.logging.path is not None
	else "/var/log/bitops"
	)

	BITOPS_plugin_dir = (
	BITOPS_ENV_plugin_dir
	if BITOPS_ENV_plugin_dir is not None
	else bitops_build_configuration.bitops.plugins.plugin_dir
	if bitops_build_configuration.bitops.plugins.plugin_dir is not None
	else "/opt/bitops/scripts/plugins/"
	)

	BITOPS_INSTALLED_PLUGINS_DIR = "/opt/bitops/scripts/installed_plugins/"

	BITOPS_default_folder = (
	BITOPS_ENV_default_folder
	if BITOPS_ENV_default_folder is not None
	else bitops_build_configuration.bitops.default_folder
	if bitops_build_configuration.bitops.default_folder is not None
	else "_default"
	)

	BITOPS_timeout = (
	BITOPS_ENV_timeout
	if BITOPS_ENV_timeout is not None
	else bitops_build_configuration.bitops.timeout
	if bitops_build_configuration.bitops.timeout is not None
	else 600

Implementing application level configuration parsing + deployment seq… #413

Implementing application level configuration parsing + deployment seq… #413

Conversation

PhillypHenning commented Mar 2, 2023 • edited by arm4b Loading

Description

Type of change

How Has This Been Tested?

Checklist:

runforesight bot commented Mar 2, 2023 • edited Loading

Foresight Summary

✅ CI workflow has finished in 35 seconds and finished at 8th Mar, 2023.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arm4b Mar 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arm4b Mar 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arm4b left a comment

Choose a reason for hiding this comment

PhillypHenning commented Mar 2, 2023 •

edited by arm4b

Loading

runforesight bot commented Mar 2, 2023 •

edited

Loading

arm4b Mar 2, 2023 •

edited

Loading

arm4b Mar 7, 2023 •

edited

Loading