Use group_vars rather than env_vars for playbook variables.

[YPCrumble]

This is the initial fix for #66.

This PR replaces the current env_vars directory with a group_vars directory as can be found in Ansible best practices as well as in the best practices for incorporating Ansible Vault.

There are (at least) three goals here:

1. Allowing the user to override any environment variables for their playbook in one place only, rather than having to search for the variable in a specific role.
2. Allowing for different variables in each group. For instance, right now it's difficult to set a separate server_user_password in both staging and production environments.
3. Allowing easy use of Ansible Vault to encrypt secrets (I would add instructions to the README if this PR gets pulled in).

@jcalazan @DavidCain @dpward @StuartMacKay do any of you have an objection to this change which would be breaking vs. the current playbook?

[DavidCain]

Hey! Thanks for doing this. I have no inherent objections, but would love to review this and read up a bit more on the trade-offs. Obviously, this would require some work from those of us who maintain forks in order to stay up to date with the project. =)

The link to Ansible best practices is interesting, and that strategy definitely solves a lot of the problems I've had with Ansible Vault. While the practice of keeping two files seems sensible, I've been using another standard strategy to encrypt some variables that keeps everything grepable and visible in a Git history. I use Ansible's encrypt_string, which has output like the following:

some_foo: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      62313365396662343061393464336163383764373764613633653634306231386433626436623361
      6134333665353966363534333632666535333761666131620a663537646436643839616531643561
      63396265333966386166373632626539326166353965363262633030333630313338646335303630
      3438626666666137650a353638643435666633633964366338633066623234616432373231333331
      6564

In short, encrypt_string allows you to encrypt just some content, not the whole file. It leaves you free to include comments or other non-encrypted content alongside the encrypted secrets.

Over in my fork of this project, I use this strategy for managing production secrets: https://github.com/DavidCain/mitoc-ansible/blob/master/env_vars/production.yml (anywhere in that file with the string "REDACTED" was formerly an encrypted secret, before I used filter-branch to scrub the repo).

[YPCrumble]

@DavidCain thanks for sharing! Offhand - does encrypt_string require you to encrypt every string separately?

In terms of searchability, here's the strategy with ansible-vault (which I'd include in the updated README with instructions on using ansible-vault):

Your group_vars/[group_name]/ directory has two files - vars.yml and vault.yml. The vars.yml includes any vars that don't require encryption AND all of your encrypted variables from vault.yml. The vault.yml file includes your encrypted variables, prefixed with vault_.

In this way, you can easily find variables that are encrypted by grepping in vars.yml, and you can encrypt all your variables in one go in vault.yml.

Example vault.yml and vars.yml files:

vars.yml

first_unencrypted_variable: "This is the first unencrypted variable."
second_unencrypted_variable: "This is the second unencrypted variable."

first_encrypted_variable: "{{ vault_first_encrypted_variable }}"
second_encrypted_variable: "{{ vault_second_encrypted_variable }}"

vault.yml (decrypted)

vault_first_encrypted_variable: "This is the first encrypted variable."
vault_second_encrypted_variable: "This is the second encrypted variable."

[DavidCain]

Yup, encrypt_string does require individual encryption of secrets. Of course, you can always use shell scripts and such, but it's not like Vault where you can just edit an entire file and decrypt/encrypt with one operation.

[YPCrumble]

@DavidCain got it. I'm still inclined to move in the direction of this PR because (for me at least) encrypting each secret one at a time would take a really long time. Making it so that there's only one file to edit (two with a vault file) to deploy a new instance is another goal that I think this brings us closer to.

In any case, take your time to go through it and read up on the trade-offs like you said. If there's a better solution I'd be happy to do that!

[DavidCain]

That line of thinking makes perfect sense to me, and it's backed up with a standard Ansible pattern! I'll go through this diff soon when I have some time, but I think your approach is very reasonable.

[YPCrumble]

@jcalazan @DavidCain @dpward thoughts on merging this PR in? It's backwards incompatible but in-line with Ansible best practices.

I'm currently using a forked version with this setup for my own deployments and it's working well; for me as a maintainer merging this in would make updates a lot smoother, but I don't want to be selfish if any of you has very strong objections.

[isedwards]

Would it make sense for the project to adopt semver and resume making occasional releases?

Happy for you to make backwards incompatible changes any time you need to as long as it's very obvious to everyone.

[YPCrumble]

@isedwards that makes a lot of sense. I'd propose releasing the current version as 1.2 and then merging this PR would be v2.0. I would also include the summary of the breaking change in the release notes.

[isedwards]

Perfect. Perhaps you could then do a v2.1 either when
a) there's something significant that's really worth updating to (esp. security fixes), or
b) about 6 months has past

People like me can then stick with the releases rather than being on a competing random commit for a random amount of time... (I've got no real feeling for how old the commit I'm currently using is, or whether there's been anything really significant I should be updating to).

[DavidCain]

Just wanted to chime in that I'm in support of adopting this change and using semver to indicate major changes to the repo. +1 to Ian's thoughts - it might be nice to lump in some other major breaking change (like dropping support for Ubuntu 16.04, which loses LTS support in April 2021).

[YPCrumble]

@StuartMacKay thoughts on this PR? I believe this is the canonical way to manage multiple sites under one playbook. I also know it potentially replaces one of your PR's. I'm aiming to release this as a 2.0 version of this repo but want to make sure others are on board!

[StuartMacKay]

I like that this is inline with best practices and is vault friendly. This is a good playbook anything that makes it stronger is definitely OK by me.

When I started using ansible-django-stack it was my first foray with ansible, before that it was Fabric. Initially I wrestled with it a bit as my project layouts were sufficiently different from the layout here that I ended up trying to redefine everything. Eventually I just stuck everything I need to override in a top-level json (now yaml) file. Which I include when running the playbook:

ansible-playbook -i projects/mysite site.yml --extra-vars="@projects/mysite.yml" --ssh-extra-args=-A --ask-become-pass

This gives me complete control and it allows me to use one checked out version of the playbook which I can use with other projects. More importantly it means I can also use the same playbook for development. There's no risk I end up committing deployment related changes.

So this last point is my only reservation over group_vars. I'd really rather leave the playbook source tree completely alone and not have my site related stuff laying around in it. However I probably don't know very much about how group_vars work and I can probably still use my "master override" yaml file.

tl;dr; go for it. I can adapt to any change.

[YPCrumble]

@StuartMacKay great point - and your "master override" should be able to work fine using the group_vars syntax.

I'm going to merge this in and we'll be at v2.0. I've also updated most of the underlying libraries and gotten tests working again.