Proposal: Deploy packs using StackStorm instead of CircleCI

[blag]

Sometimes we have to update the CircleCI configurations for all Exchange packs, and this is very tedious since our scripts have to wait for the deploy step of the CircleCI run for each pack to finish before merging the next PR or branch and kicking off another CircleCI run.

Some examples of where this is required:

• Proposal: Stop caching anything with CircleCI for Exchange packs #22
• Adding apt update to bust the apt cache for the deploy step
• Add and declare Python 3 support to all Exchange packs
• Block deployment on both Python 2 and Python 3 tests passing
• Update requests version
• Remove Python 2 support (in the future)

[arm4b]

Yes, there are problems with the missing locking and synchronization for the index file when building many packs in parallel which could be improved.

However it's important the way how StackStorm-Exchange was built, open, transparent and self-sufficient based on free CircleCI and requiring no additional infrastructure. Additionally,
Anyone could just clone our system and enable CircleCI without the need to spin-up a dedicated StackStorm instance for it and avoiding complexity.

[arm4b]

Talking about the Deployment and index synchronization issue for StackStorm-Exchange,
looking at the CircleCI config this could be improved so we minimize the problem making races really rare to hit.

Here is how to do that:

Minimize the Exchange index conflict

• this is achievable by making the index update taking minimum possible time
• We should re-download index on deployment step before re-generating it updating/pushing it. This means the time window when the race/conflict possible is around second or so.
• pushing outdated index should result in a git conflict and deployment failure, which should be safely restarted, see below

Make deployment operations idempotent

• write all the steps in deployment step so they're fully idempotent
  • if deploy failed due to index conflict, - re-starting the deployment step will lead to retry for the failed operation, skipping all the previous steps that were successful
  • re-run the deployment step deploy || deploy to minimize human if deployment failed due to index conflict (race) if other pack already updated the index

This way we can minize the StackStorm-Exchange pack deployment issues up to state when human interruption is almost not needed. Some of those steps are already in-place in the current CircleCI Deployment, so this should be fairly simple change and a quick win.

[arm4b]

Alternative conflict-free approach: deploy isolated per-pack index update

• instead of every pack pushing the change to main index.json, let them push isolated per-pack index meta, there is already dir for that: https://github.com/StackStorm-Exchange/index/tree/master/v1/packs
• Setup every minute CircleCI cron job that will iterate over the pack dirs with their meta indexes and generate main StackStorm-Exchange index if there is a diff.

^^ This will require more work comparing to previous enhancement, but this approach removes the possibility of index conflict at all.

So there are plenty of options making things simple, working and self-sufficient, instead of relying on external StackStorm or lock-control instance.

[cognifloyd]

We have moved Exchange CI to GHA away from CircleCI, so I think we can close this.

[arm4b]

Good call, agreed 💯