Fix dependency conflicts

[blag]

This PR backs off on some of the updates to dependencies, and makes make requirements run successfully, on both Linux and Mac OS X (I tested both). This should also hopefully fix the weekly pack Circle CI runs.

I also tweak the requirements header to include directions on how to actually update dependencies, since it's definitely not a straightforward process.

On top of that, I also make sure the make targets actually fail, so (hopefully) future updates that break things won't pass CI.

Closes #4770 and #4771.
Reverts parts of #4689 and #4767.

[m4dcoder]

@blag I'm not sure I understand the PR here. The title says backoff some of the dependency changes but the commits here are upgrading the dependencies. Can you provide some more context?

[blag]

Sure, so the originating context here is that a few of the weekly pack CI runs have been failing and I took it upon myself to fix them if I could.

One of them was the stackstorm-git pack, which failed with this error:

error: amqp 2.5.0 is installed but amqp<3.0,>=2.5.1 is required by set(['kombu'])

@Kami had updated the dependency version of amqp to 2.5.0 in #4767, which also updated kombu to v4.6.4, which in turn requires amqp>=2.5.1,<3.0.

So #4767 broke anything that tried to install the dependencies in requirements.txt (like the pack CI), because pip could not satisfy two opposing constraints.

So I upgraded the required amqp version to 2.5.1, and then needed to run make requirements to put that change into each ST2 component requirements.txt, but that failed due to further issues.

At some point I ran across the error where pip had installed PyYAML v5.1.2, but the pack constants still expected pyyaml>=3.12,<4.0, so I upgraded that. PyYAML versioning has been a bit chaotic in the past few months, but that's a whole 'nuther story altogether.

The requests library upgrade was good, except that prance v0.6.1 required requests~=2.17 and we updated to use version 2.22.0, so that package install failed until I upgraded it to version 0.15.0 and downgraded requests back to 2.21.0, since prance still pins requests to that version.

The next failure I came upon was where six==1.12.0 was installed in the fixed-requirements.txt, but the Makefile explicitly installed version 1.11.0, so I tweaked the makefile to re-install six version 1.12.0.

I wasn't done with dealing with that section of the Makefile though, because immediately after that it uninstalls pytz, python-dateutil, and orquesta, and then runs pip install on requirements.txt and test-requirements.txt. When pip tries to install logshipper, it expects python-dateutil to already have been installed, so pip fails. The only clue I had there was the comment "Fix for Travis CI caching issue", so I threw the pip uninstalls behind a conditional that checks if it's running on Travis. If it is, it uninstalls them, and if it isn't (eg: if I'm just running make requirements locally), it leaves the packages installed so installing logshipper succeeds.

I also added instructions on how to update the dist_utils.py files and the requirements.txt files that we generate.

I've updated the title of this PR to more accurately reflect what it's attempting to achieve.

[blag]

It's not just the weekly pack CI runs that are broken, it's also pack PR CI jobs that are broken:
StackStorm-Exchange/stackstorm-zabbix#35
https://circleci.com/gh/StackStorm-Exchange/stackstorm-zabbix/189

[arm4b]

👍 The good, - it closes #4770
👎 The bad - it re-opens/reverts #4680 for which we advertised in Roadmap that we support latest requests.

I'm also unsure why do we need Mac OS X build and what consequences it brings. It was discussed several times before that we don't support any OSX-related builds and make sure tries to do that don't affect any st2 code.
If the rabbit hole is OS X related, I'd better go with more minimalistic idea behind the #4771 fix.

[blag]

I would rather have a roadmap bullet point slip than break all pack CIs.

ST2 still doesn't build on OS X (and likely never will), but at least we can run make requirements on it and have it generate the requirements files.

[arm4b]

From the rabbit hole investigation you described in detail #4774 (comment), how the resulting code would differ in this PR without OS X requirements, eg. if making requirements on OS we officially support?

[blag]

The only pieces that are Mac OS X support are:

• the new scripts/write-headers.sh script
• the code that now calls that write-headers.sh

And even with this change, we still only officially support building on Linux. The build still ultimately fails on OS X, since the Python inotify package doesn't successfully install on anything but Linux.

I did make sure that make requirements successfully completed on Linux, and that it didn't make any changes from when I ran it on OS X.

[arm4b]

After reviewing requests Changelog, I'm fine with downgrading it to previous minor version 👍

Please include Changelog record for PIP/requirements fixes you've made.

[arm4b]

Please include 1 bugfix Changelog record for this PR instead of 3, make it brief and clear.

[m4dcoder]

@blag I appreciate the effort you put into getting all of this to work. I'm OK w/ the fixes here. The reason with my delayed response here is because the changes are a mishmash of dependency fixes. I have to reread your explanation several times. I came to the conclusion here that this fixes a number of dependency issues which plagues exchange CI. It'll be unproductive to unnecessary block this. Maybe we do need a better strategy to manage dependency. But until we identify a solution, I think it will be helpful in the future for all of us to be very succinct about grouping changes and keeping in scope. For example, OSX is not a supported distro. For efficiency sake, the recommended platform for developers should be on Linux. I see little value in maintaining whatever compatibility or functionality on OSX.

[m4dcoder]

@armab I'm Ok w/ the fixes here as explained above. I'll let you do the honor to approve this PR since you're still working on some minor details with @blag.

[blag]

Resolved all requested changes.

[arm4b]

LGTM,
Thank You for fixing Stackstorm-Exchange CI 👍