Use Python to generate .travis.yml

[Eric-Arellano]

Problem

While travis.yml.mustache is a big improvement over what we had a year ago—before we used Mustache templates and anchors which prevented any sort of deduplication—the file is difficult to understand. YAML anchors are tough to track, especially when you start using multiple anchors like we currently do. Likewise, the Mustache files are better than what we had before but require jumping to multiple files to see the result.

There is no particular reason we are using YAML + Mustache to generate the final .travis.yml. While the final config file must be in YAML format for Travis to understand it, there is no requirement on what language we use to actually generate the file, so long as it ends up being .travis.yml.

Goal

1. The generated .travis.yml must be functionally identical to the current one, even if the actual YAML files differs. What does this mean? YAML is ultimately a superset of JSON. To check that the current version is identical to the new version, you can run both through https://yamlvalidator.com, which will convert the YAML into JSON, applying things like anchors for you. We need the JSON to resolve the same for the current version and whatever final version we end up with.
2. We should stop using anchors / inheritance in the generated .travis.yml. These make it very difficult to understand, and were only used to deduplicate things. We don't care if the generated .travis.yml ends up being 5k lines long due to duplication - more important is that you can go to any individual Travis job and see all of its config in one spot.
3. The file used to generate should be readable and maintainable.

Suggestion: Use PyYAML

PyYAML allows us to use readable Python to generate a YAML file via its safe_dump() function.

We only pass PyYAML the dictionary we want. For example:

safe_dump({
  "conditions": "v1",
  "env": {"global": ["PANTS_CONFIG_FILES=...", ...]},
   ...
})

How is this any better? We can use Python's control flow and abstractions to deduplicate things and make them more readable. For example, we can use functions + dictionary unpacking via ** to define an entry like this:

unit_tests_v2_py36 = {**linux_shard(python_version="3.7", include_test_config=True), **unit_tests_v2_base, "my_special_config": "test"}

[Eric-Arellano]

See #8163 for an initial proof of concept. @gshuflin feel free to build off of this - know you're on vacation right now. I gotta get back to teaching the next few days haha.

[cosmicexplorer]

This seems like a great idea!!! Mustache is the lowest common denominator sometimes but being able to introduce more logic in the language we love to use seems like a big win.