Custom pack deps script

[cognifloyd]

I need some additional setup to run tests in the StackStorm-Exchange/stackstorm-vault#19.

This adds running the custom testing env setup script to the dependencies script.
It also adds a sample (empty) script and bootstraps new packs with it.

[blag]

I'd like one more review on this before we merge it.

I just noticed something weird: no pack has a requirements-tests.txt file, but the ManageIQ pack has a test-requirements.txt file even though it's Makefile references requirements-tests.txt, and the Network Essentials pack (which I will be removing once StackStorm v3.5 is released) references a requirements-tests.txt file but doesn't have one. ¯\_(ツ)_/¯

[cognifloyd]

For background:

This PR makes the dependencies script run <pack>/tests/setup_testing_env.sh if present and executable.
The dependencies script is run as one of the steps in the circleci pack workflows. For python3, that step looks like this:

            git clone -b master git://github.com/stackstorm-exchange/ci.git ~/ci
            ~/ci/.circle/dependencies ; ~/ci/.circle/exit_on_py3_checks $?

The next step actually runs the pack tests. For python3, that is: ~/ci/.circle/test ; ~/ci/.circle/exit_on_py3_checks $?

The alternative to including this would be have each pack that needs this edit .circleci/config.yml to add those custom steps. But, doing that would make maintaining all the packs on the exchange much more difficult, as there would be a much larger delta making ci infrastructure upgrades very difficult.

[arm4b]

The alternative to including this would be have each pack that needs this edit .circleci/config.yml to add those custom steps. But, doing that would make maintaining all the packs on the exchange much more difficult, as there would be a much larger delta making ci infrastructure upgrades very difficult.

Right. I don't remember where, but I think some packs already did that, for instance installed OS-level dependencies by modifying CircleCI config. I think that's OK approach, but may come with a price you mentioned.
From the other side not sure if adding setup_testing_env.sh magic script would be something known and widely used and eventually we may end up with 2 approaches being deployed depending on pack maintainer(s).

[cognifloyd]

The feature is "advertised" by adding it by default to new packs. And, for all packs that are currently editing circle config, we could migrate them to use this script.

I'm ready/willing to prepare/push PRs for all the exchange packs to drop python2.7 support: StackStorm/community#40 (comment)

Either as part of that PR, or a separate one if we can stand the noise of multiple PRs across all exchange packs, that standardizes .circleci/config.yml for all packs by moving all of the customization into this script. That would probably include a warning at the top of the config file that says it something like:

# WARNING: Minimize edits to this file!
# 
# This file is part of the CI infrastructure for the StackStorm-Exchange.
# As such, it gets overwritten periodically in CI infrastructure updates.
# Check out `tests/setup_testing_env.sh` for how to customize the test env.
# If you need to add jobs, docker images, or other changes that do not work
# in `tests/setup_testing_env.sh`, then please add what you need and avoid
# changing the standard build_and_test and deploy jobs.
#
# Thanks for your contribution!
---

[cognifloyd]

This is where I'm adding the script by default during pack bootstrap:
https://github.com/StackStorm-Exchange/ci/pull/101/files#diff-05f1a400c825f9abf994a07572f784b9f8e9ac708c650dfdd808817320c156d9

[arm4b]

Even with the added magic script setup_testing_env.sh, there's nothing that would prevent pack maintainers from modifying the CircleCI config and that sounds OK to me.

For instance, I see a point if users might need to add an external service available for them for some integration test scenarios. This could be done in a native and consistent to CircleCI way by defining new Docker images which is OK and not always possible with the script:

  build_and_test_python36:
    docker:
      - image: circleci/python:3.6
      - image: rabbitmq:3
      - image: mongo:3.4
      - image: service1 # added
      - image: service2 # added

https://github.com/StackStorm-Exchange/stackstorm-vault/blob/f809a6276d5ff2d79d1a7074daa691ff91f4d590/.circleci/config.yml#L49-L53

[cognifloyd]

This is the conversation (view the thread) that sent me down this path:
https://stackstorm-community.slack.com/archives/CSBELJ78A/p1611207453003200

Jacob Floyd Jan 20th at 23:37

For pack development on stackstorm-exchange, do we ever overwrite .circleci/config.yml across all packs? Or is it ok to edit .circleci/config.yml to install additional system deps for end-to-end tests in a pack?

blag 8 days ago

We do have some tooling to modify all CircleCI configs across all packs [1], and those tools will be very useful to us once we deprecate Python 2 in ST2 Exchange. So try to minimize the changes you make to the config file.

But otherwise, if you need to install system dependencies, feel free to tweak the config file to do so. If this is something that we should offer to all packs, maybe consider tweaking the master .circle/config.yaml.sample template [2] in the StackStorm-Exchange/ci repo to add an empty Install system dependencies step, and/or tweak the dependencies script to handle system dependencies [3].

[1] https://github.com/StackStorm-Exchange/ci/blob/master/tools/trigger_circle_ci_builds_for_all_packs.sh,
https://github.com/StackStorm-Exchange/ci/blob/master/tools/clone_all_repos.sh
StackStorm-Exchange/exchange-tools#2
[2] https://github.com/StackStorm-Exchange/ci/blob/master/.circle/circle.yml.sample
[3] https://github.com/StackStorm-Exchange/ci/blob/master/.circle/dependencies

Jacob Floyd 8 days ago

Hmm. That's a good idea. Maybe a separate script that the circle config calls if present, and that script can do whatever test setup is required. Then it can really be the same across all repos. Cool. I'll figure out what that looks like with vault, and then submit a pr against the ci repo.

blag 8 days ago

Keep in mind that if you eloquently handle different package names across different distributions (eg: libsomething-devel on RHEL-based and libsomething-dev on Debian-based), and across distro versions (eg: gnupg pointed to gnupg1 on Ubuntu 16.04, but it pointed to gnupg2 on 18.04+), we can then use that to have StackStorm automatically install system dependencies when packs are installed.

One thing to look out for is that you want to avoid creating more work for us, so you don’t want to force us to update these dependency files every time a new distro or distro version is released. So if an OS is not explicitly specified, try to have some logic find the latest packages to install.

Example:

Ubuntu:  # Distro name
  # For Ubuntu 20.04, have the system-dependency script implicitly try to install the dependency names from Ubuntu 18.04, or whatever is deemed to be the "latest" version
  - "18.04":  # Version specifiers are *strings*! If you do not quote them, YAML will interpret them as floats, which can cause subtle, unpredictable bugs when comparing version strings!
      # Recommend specifying all requirements in alphabetical order, just to make it easier to compare dependency lists across versions
      - gnupg2
      - libsomething-dev  # Make each distro version completely specify which packages are necessary
  - "16.04":
      - gnupg1
      - libsomething-dev
RHEL:
  - "8":
      - gnupg
      - libsomething-devel
  - "7":
      - gnupg
      - libsomething-devel

Jacob Floyd 8 days ago

For vault, there won't actually be sys deps per say, at least no rpm/Deb to install. I'll download the git repo for hvac (underlying python lib) and use their test script to download the vault app directly and use that for tests. So in this case, it should be fairly portable, but it's only needed for testing. (edited)

blag 8 days ago

Oh, gotcha. Yeah in that case you might just tweak the dependencies script to call a install-custom-dependencies.sh script and create a blank/template install-custom-dependencies.sh script so other users know how to use it:

#!/bin/bash

# This script runs any custom commands necessary to setup the testing environment
# Use requirements.txt and/or test-requirements.txt to install Python package dependencies and Python test package dependencies, respectively.

[blag]

@armab Those are good points.

If given a choice between the two options of:

1. customizing circleci/config.yml for different packs
2. customizing a hook for different packs

then I would choose option 2.

I think the original intent behind exchange-incubator was to keep the CircleCI configs as consistent across packs as possible. While some packs may need to customize them, every change makes it more difficult for us to make the occasional sweeping changes across all packs, like removing the Python 2.7 test workflow, or add or updating a Python 3.x workflow in the future.

I think adding support for a hook that is outside of the CircleCI config is a win for us.

[blag]

@armab does have a valid point wrt adding Dockerizable/Dockerized service to the config. To a certain extent, customizing CircleCI configs a little bit will be unavoidable. But adding a mechanism to keep some customizations out of that file is still beneficial.

[arm4b]

Thanks for the pointers. Yeah, it might be a good hook addition indeed, but we can't force users to use only that mechanism.

Side Note: when removing Python2 job from all the CircleCI configs would be nice to rely on a smarter parsing primitives, rather then just pushing all CircleCI configs to one format.

[cognifloyd]

Ooh. Two 👍 reviews. Could someone merge this please?

[cognifloyd]

This is an example PR for what I'd like to do to all exchange packs after this is merged:
StackStorm-Exchange/stackstorm-acos#18

[cognifloyd]

Good news: no migrations are required to use this new script in any of the packs.
Bootstrapping it across packs would be helpful to make the new feature visible.

I just looked at the circleci config in all the packs to:

1. see if there are any significant customizations that could use this new script,
2. see what the difference between the default circleci config and the pack's circleci config, and
3. prepare an update for all packs that is unrelated to this PR (drop python2.7 tests).

The zabbix pack is the only pack that has significant customization of the circle-ci config. Happily, the zabbix pack only adds some jobs, it doesn't modify the default jobs, so dropping the python2.7 test job is a straight forward merge.

The only other notable difference in the packs' circleci config was the weekly schedule: some on Saturday, some on Sunday.

(And the test packs' config is significantly out-of-date).

[cognifloyd]

I added this warning to .circle/circle.yml.sample:

# WARNING: Minimize edits to this file!
#
# This file is part of the CI infrastructure for the StackStorm-Exchange.
# As such, it gets overwritten periodically in CI infrastructure updates.
# Check out `tests/setup_testing_env.sh` for how to customize the test env.
# If you need to add jobs, docker images, or other changes that do not work
# in `tests/setup_testing_env.sh`, then please add what you need and avoid
# changing the standard build_and_test and deploy jobs.
#
# Thanks for your contribution!

[cognifloyd]

Sorry for the commit reference spam in here.

Note to self. Do not use URLs in commit messages when working on exchange-wide changes.

[cognifloyd]

Can we merge this?
I suspect the exchange needs a revamped CI that somehow include Github Actions (and perhaps modulesync or some other yaml/templating utilities) to keep CI config up-to-date, with plenty of ways to customize the workflows per repository. I will start drafting an RFC to post on StackStorm/discussions. But for now, can we merge this, please?