Merge pull request #107 from cfstacks/cleanup

Cleanup

docs/source/guides/step_by_step_walkthrough.rst

@@ -403,14 +403,14 @@ We are going to be deploying stacks using ``project1`` profile. If you prefer
 using environment variables for configuring AWS authentication, then you
 don't have to specify the profile.
 
-Stacks requires you to provide an environment name, for this guide, let's call
+Sample templates requires you to provide an environment name, for this guide, let's call
 it ``learning``. Properties file is environment-aware and can have a common set
 of properties as well as per-environment ones. See :doc:`/fundamentals/configuration`.
 
 There are two ways to set an environment:
 
 * ``STACKS_ENV`` environment variable
-* ``--env`` command line argument to the subcommands
+* ``--env`` or ``-e`` command line argument to the subcommands
 
 Create the VPC stack:
 

docs/source/quickstarts/getting_started.rst

@@ -55,18 +55,18 @@ Create a template file named ``buckets.yaml``:
 
 .. code-block:: jinja
 
-  ---
-  name: s3-buckets
+    ---
+    name: s3-buckets
 
-  ---
-  AWSTemplateFormatVersion: '2010-09-09'
-  Description: S3 buckets
-  Resources:
+    ---
+    AWSTemplateFormatVersion: '2010-09-09'
+    Description: S3 buckets
+    Resources:
     {% for n in range(buckets_count|int) %}
     S3Bucket{{ loop.index0 }}:
       Type: AWS::S3::Bucket
       Properties:
-        BucketName: {{ bucket_name_prefix }}{{ loop.index0 }}-{{ region }}
+        BucketName: {{ env }}-{{ bucket_name_prefix }}{{ loop.index0 }}-{{ region }}
     {% endfor -%}
 
 In summary, the template describes, that ``s3-buckets`` stack contains
@@ -77,34 +77,34 @@ defined via ``buckets_count`` property.
 Deploy your template
 --------------------
 
-Create our first stack, ignore ``--env tutorial`` argument for now:
+Create our first stack:
 
 .. code-block:: shell
 
-  $ stacks create --env tutorial --template buckets.yaml
+  $ stacks create --template buckets.yaml --env dev
   Required properties not set: buckets_count,bucket_name_prefix
 
 We get an error about missing properties. To fix that, we must specify the
 missing properties. Add the ``--follow`` flag, to follow stack events:
 
 .. code-block:: shell
 
-  $ stacks create --env tutorial --template buckets.yaml --follow \
+    $ stacks create --template buckets.yaml --env dev --follow \
       --property bucket_name_prefix=my-awesome-bucket \
       --property buckets_count=3
-
-  2015-12-29 15:04:26.358000  CREATE_IN_PROGRESS  AWS::CloudFormation::Stack  s3-buckets  User Initiated
-  2015-12-29 15:04:41.654000  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket0
-  2015-12-29 15:04:42.491000  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket2
-  2015-12-29 15:04:44.724000  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket0  Resource creation Initiated
-  2015-12-29 15:04:45.705000  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket1
-  2015-12-29 15:04:47.078000  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket1  Resource creation Initiated
-  2015-12-29 15:04:48.484000  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket2  Resource creation Initiated
-  2015-12-29 15:05:05.999000  CREATE_COMPLETE  AWS::S3::Bucket  S3Bucket0
-  2015-12-29 15:05:08.497000  CREATE_COMPLETE  AWS::S3::Bucket  S3Bucket1
-  2015-12-29 15:05:11.375000  CREATE_COMPLETE  AWS::S3::Bucket  S3Bucket2
-  2015-12-29 15:05:12.921000  CREATE_COMPLETE  AWS::CloudFormation::Stack  s3-buckets
-
+    2018-09-10 18:30:08.428000+01:00  CREATE_IN_PROGRESS  AWS::CloudFormation::Stack  s3-buckets  User Initiated
+    2018-09-10 18:30:10.883000+01:00  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket0
+    2018-09-10 18:30:10.963000+01:00  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket2
+    2018-09-10 18:30:10.966000+01:00  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket1
+    2018-09-10 18:30:11.732000+01:00  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket2  Resource creation Initiated
+    2018-09-10 18:30:11.837000+01:00  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket0  Resource creation Initiated
+    2018-09-10 18:30:11.923000+01:00  CREATE_IN_PROGRESS  AWS::S3::Bucket  S3Bucket1  Resource creation Initiated
+    2018-09-10 18:30:32.608000+01:00  CREATE_COMPLETE  AWS::S3::Bucket             S3Bucket2
+    2018-09-10 18:30:32.615000+01:00  CREATE_COMPLETE  AWS::S3::Bucket             S3Bucket0
+    2018-09-10 18:30:32.782000+01:00  CREATE_COMPLETE  AWS::S3::Bucket             S3Bucket1
+    2018-09-10 18:30:34.229000+01:00  CREATE_COMPLETE  AWS::CloudFormation::Stack  s3-buckets
+
+Use of ``--env`` is optional, however it's a good practice to separate resources by environment.
 
 See your new deployment
 -----------------------
@@ -113,17 +113,17 @@ See the status of your new stack by running:
 
 .. code-block:: shell
 
-  $ stacks list
-  s3-buckets  CREATE_COMPLETE
+    $ stacks list
+    s3-buckets  CREATE_COMPLETE
 
 If you want to see what resources have been created by the stack, you can do that by running:
 
 .. code-block:: shell
 
-  $ stacks resources s3-buckets
-  S3Bucket0  my-awesome-bucket0-us-east-1  AWS::S3::Bucket  CREATE_COMPLETE
-  S3Bucket1  my-awesome-bucket1-us-east-1  AWS::S3::Bucket  CREATE_COMPLETE
-  S3Bucket2  my-awesome-bucket2-us-east-1  AWS::S3::Bucket  CREATE_COMPLETE
+    $ stacks resources s3-buckets
+    S3Bucket0  dev-my-awesome-bucket0-us-east-1  AWS::S3::Bucket  CREATE_COMPLETE
+    S3Bucket1  dev-my-awesome-bucket1-us-east-1  AWS::S3::Bucket  CREATE_COMPLETE
+    S3Bucket2  dev-my-awesome-bucket2-us-east-1  AWS::S3::Bucket  CREATE_COMPLETE
 
 
 Clean up
@@ -135,13 +135,11 @@ To delete the deployment, run:
 
 .. code-block:: shell
 
-  $ stacks delete -y s3-buckets
+    $ stacks delete s3-buckets -y
 
 
 Next steps
 ----------
 Now that you have an idea of how stacks enhances CloudFormation, we recommend
 going through :doc:`/guides/step_by_step_walkthrough` for more comprehensive
 walkthrough.
-
-

stacks/__about__.py

@@ -1,4 +1,4 @@
-__version__ = '0.4.1'
+__version__ = '0.4.2'
 __licence__ = 'MIT'
 __url__ = 'https://stacks.tools'
 __maintainer__ = 'Vaidas Jablonskis'

stacks/cf.py

@@ -4,26 +4,25 @@
 # An attempt to support python 2.7.x
 from __future__ import print_function
 
-import sys
 import builtins
-import time
-import yaml
-import json
-import jinja2
 import hashlib
+import json
+import sys
+import time
+from datetime import datetime
+from fnmatch import fnmatch
+from operator import attrgetter
+from os import path
+
 import boto
+import jinja2
 import pytz
 import tzlocal
-
-from os import path
+import yaml
+from awscli.customizations.cloudformation.yamlhelper import intrinsics_multi_constructor
+from boto.exception import BotoServerError
 from jinja2 import meta
-from fnmatch import fnmatch
 from tabulate import tabulate
-from boto.exception import BotoServerError
-from operator import attrgetter
-from datetime import datetime
-
-from awscli.customizations.cloudformation.yamlhelper import intrinsics_multi_constructor
 
 from stacks.aws import get_stack_tag
 from stacks.aws import throttling_retry
@@ -302,8 +301,8 @@ def print_events(conn, stack_name, follow, lines=100, from_dt=datetime.fromtimes
     while True:
         events, next_token = get_events(conn, stack_name, next_token)
         status = get_stack_status(conn, stack_name)
+        normalize_events_timestamps(events)
         if follow:
-            normalize_events_timestamps(events)
             events_display = [(ev.timestamp.astimezone(tzlocal.get_localzone()), ev.resource_status, ev.resource_type,
                                ev.logical_resource_id, ev.resource_status_reason) for ev in events
                               if ev.event_id not in seen_ids and ev.timestamp >= from_dt]
@@ -315,8 +314,8 @@ def print_events(conn, stack_name, follow, lines=100, from_dt=datetime.fromtimes
             if next_token is None:
                 time.sleep(5)
         else:
-            events_display.extend([(event.timestamp, event.resource_status, event.resource_type,
-                                    event.logical_resource_id, event.resource_status_reason)
+            events_display.extend([(event.timestamp.astimezone(tzlocal.get_localzone()), event.resource_status,
+                                    event.resource_type, event.logical_resource_id, event.resource_status_reason)
                                    for event in events])
             if len(events_display) >= lines or next_token is None:
                 break

stacks/cli.py

@@ -1,4 +1,5 @@
 import os.path
+
 import configargparse
 
 from stacks import __about__
@@ -15,74 +16,91 @@ def parse_options():
     parser.add_argument('--version', action='version', version=__about__.__version__)
     subparsers = parser.add_subparsers(title='available subcommands', dest='subcommand')
 
+    # resources subparser
     parser_resources = subparsers.add_parser('resources', help='List stack resources')
     parser_resources.add_argument('name', help='Stack name')
     parser_resources.add_argument('logical_id', nargs='?', default=None,
                                   help='Logical resource id. Returns physical_resource_id.')
 
+    # outputs subparser
     parser_outputs = subparsers.add_parser('outputs', help='List stack outputs')
     parser_outputs.add_argument('name', help='Stack name')
     parser_outputs.add_argument('output_name', nargs='?', default=None,
                                 help='Output name. Returns output value.')
 
+    # config subparser
     parser_config = subparsers.add_parser('config', help='Print config properties')
-    parser_config.add_argument('-e', '--env', env_var='STACKS_ENV')
+    # noinspection PyArgumentList
+    parser_config.add_argument('-e', '--env', env_var='STACKS_ENV', required=False, default=None)
     parser_config.add_argument('-o', '--output', default='text', choices=['text', 'yaml', 'json'],
                                dest='output_format', help='Output format')
+    # noinspection PyArgumentList
     parser_config.add_argument('-c', '--config', default='config.yaml',
                                env_var='STACKS_CONFIG', required=False,
                                type=_is_file)
+    # noinspection PyArgumentList
     parser_config.add_argument('--config-dir', default='config.d',
                                env_var='STACKS_CONFIG_DIR', required=False,
                                type=_is_dir)
     parser_config.add_argument('property_name', nargs='?', default=None)
 
+    # list subparser
     parser_list = subparsers.add_parser('list', help='List stacks')
     parser_list.add_argument('-v', '--verbose', action='store_true')
     parser_list.add_argument('name', default='*', nargs='?',
                              help='Stack name or unix shell-style pattern')
 
+    # create subparser
     parser_create = subparsers.add_parser('create', help='Create a new stack')
     parser_create.add_argument('-t', '--template', required=True, type=configargparse.FileType())
+    # noinspection PyArgumentList
     parser_create.add_argument('-c', '--config', default='config.yaml',
                                env_var='STACKS_CONFIG', required=False,
                                type=_is_file)
+    # noinspection PyArgumentList
     parser_create.add_argument('--config-dir', default='config.d',
                                env_var='STACKS_CONFIG_DIR', required=False,
                                type=_is_dir)
     parser_create.add_argument('name', nargs='?', default=None)
-    parser_create.add_argument('-e', '--env', env_var='STACKS_ENV', required=True)
+    # noinspection PyArgumentList
+    parser_create.add_argument('-e', '--env', env_var='STACKS_ENV', required=False, default=None)
     parser_create.add_argument('-P', '--property', required=False, action='append')
     parser_create.add_argument('-d', '--dry-run', action='store_true')
     parser_create.add_argument('-f', '--follow', dest='events_follow', help='Follow stack events', action='store_true')
 
+    # update subparser
     parser_update = subparsers.add_parser('update', help='Update an existing stack')
     parser_update.add_argument('-t', '--template', required=True, type=configargparse.FileType())
+    # noinspection PyArgumentList
     parser_update.add_argument('-c', '--config', env_var='STACKS_CONFIG',
                                default='config.yaml', required=False,
                                type=_is_file)
+    # noinspection PyArgumentList
     parser_update.add_argument('--config-dir', default='config.d',
                                env_var='STACKS_CONFIG_DIR', required=False,
                                type=_is_dir)
     parser_update.add_argument('name', nargs='?', default=None)
-    parser_update.add_argument('-e', '--env', env_var='STACKS_ENV', required=True)
+    # noinspection PyArgumentList
+    parser_update.add_argument('-e', '--env', env_var='STACKS_ENV', required=False, default=None)
     parser_update.add_argument('-P', '--property', required=False, action='append')
     parser_update.add_argument('-d', '--dry-run', action='store_true')
     parser_update.add_argument('--create', dest='create_on_update',
                                help='Create if stack does not exist.',
                                action='store_true')
     parser_update.add_argument('-f', '--follow', dest='events_follow', help='Follow stack events', action='store_true')
 
+    # delete subparser
     parser_delete = subparsers.add_parser('delete', help='Delete an existing stack')
     parser_delete.add_argument('-f', '--follow', dest='events_follow', help='Follow stack events', action='store_true')
     parser_delete.add_argument('-y', '--yes', help='Confirm stack deletion.', action='store_true')
     parser_delete.add_argument('name')
 
+    # events subparser
     parser_events = subparsers.add_parser('events', help='List events from a stack')
     parser_events.add_argument('name')
     parser_events.add_argument('-f', '--follow', dest='events_follow', action='store_true',
                                help='Poll for new events until stopped (overrides -n)')
-    parser_events.add_argument('-n', '--lines', default='10', type=int)
+    parser_events.add_argument('-n', '--lines', default=100, type=int)
 
     return parser, parser.parse_args()
 

stacks/config.py

@@ -1,8 +1,9 @@
-import sys
-import os
-import yaml
 import json
+import os
+import sys
+
 import boto
+import yaml
 
 AWS_CONFIG_FILE = os.environ.get('HOME', '') + '/.aws/config'
 AWS_CREDENTIALS_FILE = os.environ.get('HOME', '') + '/.aws/credentials'
@@ -36,11 +37,10 @@ def config_merge(env, config_file=None):
 
 def list_files(dirname):
     """Return a sorted list of files from dirname"""
-    l = os.listdir(dirname)
     lf = []
     if not dirname:
         return lf
-    for f in l:
+    for f in os.listdir(dirname):
         joined = os.path.join(dirname, f)
         if os.path.isfile(joined) and joined.endswith('.yaml'):
             lf.append(joined)
@@ -74,9 +74,8 @@ def _merge(config, env):
 def _load_yaml(fname):
     try:
         with open(fname) as f:
-            y = yaml.load(f)
-            return y
-    except:
+            return yaml.load(f)
+    except (FileNotFoundError, PermissionError, yaml.YAMLError):
         return None