Formalize the existence of external (read-only) stacks

docs/config.rst

@@ -334,13 +334,49 @@ stacks that will be deployed in the environment. The top level keyword
 *stacks* is populated with a list of dictionaries, each representing a single
 stack to be built.
 
-A stack has the following keys:
+Stacks are managed by Stacker by default. They will be created and updated as
+needed using the template file or blueprint class specified. Read-only
+external stacks can also be defined if complex cross-references are needed
+involving multiple accounts or regions.
+
+The following options are available for all stacks:
 
 **name:**
  The logical name for this stack, which can be used in conjuction with the
  ``output`` lookup. The value here must be unique within the config. If no
  ``stack_name`` is provided, the value here will be used for the name of the
  CloudFormation stack.
+**stack_name:**
+ (optional) If provided, this will be used as the name of the CloudFormation
+ stack. Unlike ``name``, the value doesn't need to be unique within the config,
+ since you could have multiple stacks with the same name, but in different
+ regions or accounts. (note: the namespace from the environment will be
+ prepended to this)
+**region**:
+ (optional): If provided, specifies the name of the region that the
+ CloudFormation stack should reside in. If not provided, the default region
+ will be used (``AWS_DEFAULT_REGION``, ``~/.aws/config`` or the ``--region``
+ flag). If both ``region`` and ``profile`` are specified, the value here takes
+ precedence over the value in the profile.
+**profile**:
+ (optional): If provided, specifies the name of a AWS profile to use when
+ performing AWS API calls for this stack. This can be used to provision stacks
+ in multiple accounts or regions.
+**external**:
+ (optional): If set to true, this stack is considered read-only, will not be
+ modified by Stacker, and most of the options related to stack deployment
+ should be omitted.
+
+The following options are available for external stacks:
+
+**fqn**:
+ (optional): Fully-qualified physical name of the stack to be loaded from
+ CloudFormation. Use instead of ``stack_name`` if the stack lies in a
+ different namespace, as this value *does not* get the namespace applied to
+ it.
+
+The following options are available for managed stacks:
+
 **class_path:**
  The python class path to the Blueprint to be used. Specify this or
  ``template_path`` for the stack.
@@ -350,7 +386,6 @@ A stack has the following keys:
  working directory (e.g. templates stored alongside the Config), or relative
  to a directory in the python ``sys.path`` (i.e. for loading templates
  retrieved via ``packages_sources``).
-
 **description:**
  A short description to apply to the stack. This overwrites any description
  provided in the Blueprint. See: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-description-structure.html
@@ -363,8 +398,10 @@ A stack has the following keys:
  updated unless the stack is passed to stacker via the *--force* flag.
  This is useful for *risky* stacks that you don't want to take the
  risk of allowing CloudFormation to update, but still want to make
- sure get launched when the environment is first created. When ``locked``,
- it's not necessary to specify a ``class_path`` or ``template_path``.
+ sure get launched when the environment is first created.
+ Note: when ``locked``, it's not necessary to specify a ``class_path``
+ or ``template_path``, but this functionality is deprecated in favour of
+ ``external``.
 **enabled:**
  (optional) If set to false, the stack is disabled, and will not be
  built or updated. This can allow you to disable stacks in different
@@ -383,22 +420,6 @@ A stack has the following keys:
 **tags:**
  (optional) a dictionary of CloudFormation tags to apply to this stack. This
  will be combined with the global tags, but these tags will take precendence.
-**stack_name:**
- (optional) If provided, this will be used as the name of the CloudFormation
- stack. Unlike ``name``, the value doesn't need to be unique within the config,
- since you could have multiple stacks with the same name, but in different
- regions or accounts. (note: the namespace from the environment will be
- prepended to this)
-**region**:
- (optional): If provided, specifies the name of the region that the
- CloudFormation stack should reside in. If not provided, the default region
- will be used (``AWS_DEFAULT_REGION``, ``~/.aws/config`` or the ``--region``
- flag). If both ``region`` and ``profile`` are specified, the value here takes
- precedence over the value in the profile.
-**profile**:
- (optional): If provided, specifies the name of a AWS profile to use when
- performing AWS API calls for this stack. This can be used to provision stacks
- in multiple accounts or regions.
 **stack_policy_path**:
  (optional): If provided, specifies the path to a JSON formatted stack policy
  that will be applied when the CloudFormation stack is created and updated.
@@ -411,13 +432,20 @@ A stack has the following keys:
  option to `wait` and stacker will wait for the previous update to complete
  before attempting to update the stack.
 
-Stacks Example
-~~~~~~~~~~~~~~
+Examples
+~~~~~~~~
+
+VPC + Instances
+:::::::::::::::
 
-Here's an example from stacker_blueprints_, used to create a VPC::
+Here's an example from stacker_blueprints_, used to create a VPC and and two EC2
+Instances::
 
+
+ namespace: example
  stacks:
- - name: vpc-example
+ - name: vpc
+ stack_name: test-vpc
  class_path: stacker_blueprints.vpc.VPC
  locked: false
  enabled: true
@@ -438,6 +466,47 @@ Here's an example from stacker_blueprints_, used to create a VPC::
  - 10.128.20.0/22
  CidrBlock: 10.128.0.0/16
 
+ - name: instances
+ stack_name:
+ class_path: stacker_blueprints.ec2.Instances
+ enabled: true
+ variables:
+ SmallInstance:
+ InstanceType: t2.small
+ ImageId: &amazon_linux_ami "${ami owners:amazon name_regex:amzn-ami-hvm-2018.03.*-x86_64-gp2}"
+ AvailabilityZone: ${output vpc::AvailabilityZone0}
+ SubnetId: ${output vpc::PublicSubnet0}
+ LargeInstance:
+ InstanceType: m5.xlarge
+ ImageId: *amazon_linux_ami
+ AvailabilityZone: ${output vpc::AvailabilityZone1}
+ SubnetId: ${output vpc::PublicSubnet1}
+
+
+Referencing External Stacks
+:::::::::::::::::::::::::::
+
+This example creates a security group in VPC from the previous example by
+importing it as an external stack with a custom profile::
+
+ namespace: other-example
+ stacks:
+ - name: vpc
+ fqn: example-test-vpc
+ profile: custom-profile
+ external: yes
+
+ - name: sg
+ class_path: stacker_blueprints.ec2.SecurityGroups
+ variables:
+ SecurityGroups:
+ VpcId: ${output vpc::VpcId}
+ SecurityGroupIngress:
+ - CidrIp: 0.0.0.0/0
+ FromPort: 22
+ ToPort: 22
+ IpProtocol: tcp
+
 Targets
 -------
 

setup.py

@@ -14,7 +14,7 @@
  "awacs>=0.6.0",
  "gitpython>=2.0,<3.0",
  "jinja2>=2.7,<3.0",
- "schematics>=2.0.1,<2.1.0",
+ "schematics>=2.1,<3.0",
  "formic2",
  "python-dateutil>=2.0,<3.0",
 ]

stacker/actions/build.py

@@ -38,44 +38,6 @@ def build_stack_tags(stack):
  return [{'Key': t[0], 'Value': t[1]} for t in stack.tags.items()]
 
 
-def should_update(stack):
- """Tests whether a stack should be submitted for updates to CF.
-
- Args:
- stack (:class:`stacker.stack.Stack`): The stack object to check.
-
- Returns:
- bool: If the stack should be updated, return True.
-
- """
- if stack.locked:
- if not stack.force:
- logger.debug("Stack %s locked and not in --force list. "
- "Refusing to update.", stack.name)
- return False
- else:
- logger.debug("Stack %s locked, but is in --force "
- "list.", stack.name)
- return True
-
-
-def should_submit(stack):
- """Tests whether a stack should be submitted to CF for update/create
-
- Args:
- stack (:class:`stacker.stack.Stack`): The stack object to check.
-
- Returns:
- bool: If the stack should be submitted, return True.
-
- """
- if stack.enabled:
- return True
-
- logger.debug("Stack %s is not enabled. Skipping.", stack.name)
- return False
-
-
 def should_ensure_cfn_bucket(outline, dump):
  """Test whether access to the cloudformation template bucket is required
 
@@ -262,7 +224,7 @@ def _launch_stack(self, stack, **kwargs):
  if self.cancel.wait(wait_time):
  return INTERRUPTED
 
- if not should_submit(stack):
+ if not stack.should_submit():
  return NotSubmittedStatus()
 
  provider = self.build_provider(stack)
@@ -272,7 +234,7 @@ def _launch_stack(self, stack, **kwargs):
  except StackDoesNotExist:
  provider_stack = None
 
- if provider_stack and not should_update(stack):
+ if provider_stack and not stack.should_update():
  stack.set_outputs(
  self.provider.get_output_dict(provider_stack))
  return NotUpdatedStatus()

stacker/actions/diff.py

@@ -222,10 +222,10 @@ def _diff_stack(self, stack, **kwargs):
  if self.cancel.wait(0):
  return INTERRUPTED
 
- if not build.should_submit(stack):
+ if not stack.should_submit():
  return NotSubmittedStatus()
 
- if not build.should_update(stack):
+ if not stack.should_update():
  return NotUpdatedStatus()
 
  provider = self.build_provider(stack)

stacker/config/__init__.py

@@ -5,11 +5,11 @@
 standard_library.install_aliases()
 from builtins import str
 import copy
-import sys
 import logging
+import sys
 
-from string import Template
 from io import StringIO
+from string import Template
 
 from schematics import Model
 from schematics.exceptions import ValidationError
@@ -19,12 +19,13 @@
 )
 
 from schematics.types import (
- ModelType,
- ListType,
- StringType,
+ BaseType,
  BooleanType,
  DictType,
- BaseType
+ ListType,
+ ModelType,
+ PolyModelType,
+ StringType
 )
 
 import yaml
@@ -225,12 +226,6 @@ def process_remote_sources(raw_config, environment=None):
  return raw_config
 
 
-def not_empty_list(value):
- if not value or len(value) < 1:
- raise ValidationError("Should have more than one element.")
- return value
-
-
 class AnyType(BaseType):
  pass
 
@@ -299,7 +294,7 @@ class Target(Model):
  required_by = ListType(StringType, serialize_when_none=False)
 
 
-class Stack(Model):
+class BaseStack(Model):
  name = StringType(required=True)
 
  stack_name = StringType(serialize_when_none=False)
@@ -308,6 +303,28 @@ class Stack(Model):
 
  profile = StringType(serialize_when_none=False)
 
+ external = BooleanType(default=False)
+
+
+class ExternalStack(BaseStack):
+ fqn = StringType(serialize_when_none=False)
+
+ @classmethod
+ def _claim_polymorphic(cls, value):
+ if value.get("external", False) is True:
+ return True
+
+ def __init__(self, *args, **kwargs):
+ super(ExternalStack, self).__init__(*args, **kwargs)
+ self.external = True
+
+ def validate_fqn(self, data, value):
+ if value and data["stack_name"]:
+ raise ValidationError("At most one of `fqn` and `stack_name` must "
+ "be provided for external stacks")
+
+
+class Stack(BaseStack):
  class_path = StringType(serialize_when_none=False)
 
  template_path = StringType(serialize_when_none=False)
@@ -368,7 +385,6 @@ def validate_parameters(self, data, value):
  "dthedocs.io/en/latest/config.html#variables for "
  "additional information."
  % stack_name)
- return value
 
 
 class Config(Model):
@@ -431,7 +447,8 @@ class Config(Model):
  ModelType(Target), serialize_when_none=False)
 
  stacks = ListType(
- ModelType(Stack), default=[])
+ PolyModelType([ExternalStack, Stack]),
+ default=[], validators=[])
 
  log_formats = DictType(StringType, serialize_when_none=False)
 

stacker/context.py

@@ -5,8 +5,8 @@
 import collections
 import logging
 
-from stacker.config import Config
-from .stack import Stack
+from stacker.config import Config, ExternalStack as ExternalStackModel
+from .stack import ExternalStack, Stack
 from .target import Target
 
 logger = logging.getLogger(__name__)
@@ -153,15 +153,18 @@ def get_stacks(self):
  stacks = []
  definitions = self._get_stack_definitions()
  for stack_def in definitions:
- stack = Stack(
- definition=stack_def,
- context=self,
- mappings=self.mappings,
- force=stack_def.name in self.force_stacks,
- locked=stack_def.locked,
- enabled=stack_def.enabled,
- protected=stack_def.protected,
- )
+ if isinstance(stack_def, ExternalStackModel):
+ stack = ExternalStack(
+ definition=stack_def,
+ context=self
+ )
+ else:
+ stack = Stack(
+ definition=stack_def,
+ context=self,
+ mappings=self.mappings,
+ force=stack_def.name in self.force_stacks
+ )
  stacks.append(stack)
  self._stacks = stacks
  return self._stacks