region-aliases and parameters used

[comister]

Hi,

Just stumbled over one use-case which I think may be worth a thought.

assuming having the following stack_master.yml

region_aliases:
  testing: us-east-1
  production: us-east-1
  staging: ap-southeast-2
stacks:
  us-east-1:
    my-stack-name:
      template: my-cf-stack.json
  ap-southeast-2:
    my-stack-name:
      template: my-cf-stack.json

So with that, I thought to put parameters under parameters/testing, parameters/production and parameters/staging with the name my_cf_stack.yml.

And I was also under the assumption if I now do a stack_master apply testing my-stack-name it will pull the parameters only from testing ... but obviously due to the same region used for testing and production it will always just take 1 of the files, regardless if I use the alias for either production or testing.

Maybe there is another way to achieve that but I am not sure on that and want to point that out here.

[patrobinson]

This is being worked on in #115 but it hasn't had <3 in a while. I too would like this feature, but it's not a high enough priority for me to work on at the moment.

[comister]

@patrobinson Thanks for pointing that out ... I have forked the repo and started to examine stack_master ... the work got started on the feature but I somehow lost track what is still missing. To me it looks like that load_stacks is already wrongly behaving and/or was maybe the point were development stopped. Will try to do something about it ... not confident enough, yet.

[comister]

I have almost understood the logic there ... but its a bit strange, is it by design that the region_alias will get replaced by the region in the config ... thats also the reason why this is not working yet, the alias is not considered anymore and will simply get replaced without any further use ... which is an approach I understand considering the alias is a real alias and not more ... maybe this should be addressed differently

[comister]

Would something like this be in line with stack_master ? (it would behave as usual but if the alias itself does hold a hash it affects the way how parameters are getting resolved ... this would address my requirement, I am not 100% sure about the original requirement of adding a prefix to the stack-name)

region_aliases:
  testing:
    region: us-east-1
    explicit_alias_parameters: true
  production: us-east-1
  staging: ap-southeast-2
stacks:
  us-east-1:
    my-stack-name:
      template: my-cf-stack.json
  ap-southeast-2:
    my-stack-name:
      template: my-cf-stack.json```

[patrobinson]

As per the README, regions can be referred to by their alias:

region_aliases:
  production: us-east-1
  staging: ap-southeast-2
...
stacks:
  production:
    myapp-vpc:
      template: myapp_vpc.rb

I think the purpose of #115 and how I imagine it should work is you can have mutliple aliases using the same region:

region_aliases:
  production: us-east-1
  staging: us-east-1
...
stacks:
  production:
    myapp-vpc:
      template: myapp_vpc.rb
  staging:
    myapp-vpc:
      template: myapp_vpc.rb

These stacks would use the same template but the parameter folder lookup would be different. Production would be:

[ "parameters/myapp-vpc.yml", "parameters/us-east-1/myapp-vpc.yml", "parameters/production/myapp-vpc.yml" ]

while staging would be:

[ "parameters/myapp-vpc.yml", "parameters/us-east-1/myapp-vpc.yml", "parameters/staging/myapp-vpc.yml" ]

[comister]

@patrobinson , right thats how I understand it as well. IMO the only reason to do that would be because of the parameters. My approach mentioned above would accept a hash for the alias with a parameter which does control the behavior of how parameterfiles would be considered. The method with the prefix appears to be quite complex to achieve this. My approach would be more straightforward (IMO)

[johnf]

My 2cs here.

It looks like the project started off with the assumption that different stacks would always exist in different regions. Hence the regions being effectively the primary key.

I think many of us have a different use case. We think in terms of environments. For example I have a an Alexa based project. The bits and pieces I need only exist in us-east-1. So I need production, staging etc all to exist in us-east-1.

I think we need to turn the current model on it's head and think about being able to have multiple environments in a region as a core use case. What I'm suggesting is a pretty major change and most likely requires a 1.0 release. My thinking is a config format something along the lines of

template_compilers:
  rb: cfndsl

stack_defaults:
  tags:
    application: echohub
    
stacks:
  web:
    s3:
      template: s3.rb
    cloudfront:
      template: cloudfront.rb
  cicd:
    ec2:
      template: ec2.rb

environment_defaults:
  region: us-east-1

environments:
  prod:
    stacks:
       - web
  staging:
    stacks:
       - web
  test:
    region: eu-west-1
    stacks:
       - web
       - cicd

This would create the following stacks in us-east-1

• prod-web-s3
• prod-web-cloudfront
• stage-web-s3
• stage-web-cloudfront

In eu-west-1 it would create

• test-web-s3
• test-web-cloudfront
• test-cicd-ec2

[johnf]

I'll start putting together a proof of concept, since I'll find something like this useful now.

[johnf]

I've created the PoC and ended up with a config that looks like

template_compilers:
  rb: cfndsl                   

region_defaults:
  us-east-1:
    secret_file: staging.yml.gpg
      
stack_defaults:                
  tags:
    application: echohub       
    
environment_defaults:
  region: us-east-1
    stacks:
      s3:
        template: s3.rb          
      pipeline:
        template: pipeline.rb    
      cloudfront:                
        template: cloudfront.rb  
      acm:
        template: acm.rb         
      route53:
        template: route53.rb     

environments:
  production:
  staging:
  testing:
    region: us-west-1
    stacks:
      cicd:
        template: cicd.rb

[patrobinson]

Here's how I've since been dealing with the problem. I have a top level directory with all templates and a directory for each environment:

|-- stack_master/
      +-- templates/
             -- web.yml
      +-- production/
             -- stack_master.yml
             +-- parameters/
                    +-- us-east-1/
                           -- web.yml
      +-- staging/
             -- stack_master.yml
             +-- parameters
                    +-- us-east-1/
                           -- web.yml

This solves two problems:

• I get to define my staging and production environments in the same region
• I don't use the same stack master config (which is 400 lines at the moment) for everything, it becomes more manageable.

I plan on having a seperate folder per AWS account actually, but I like this solution because it doesn't force you into an abstraction, you get to chose whatever you want to.

[johnf]

@patrobinson Doesn't this end up with a lot of duplication?

For example for my current project my production, staging and uat environments are all identical. With what I'm suggesting you can have a single stack_master.yml which looks like

stack_defaults:                
  tags:
    application: echohub       
    
environment_defaults:
  region: us-east-1
  stacks:
      s3:
        template: s3.rb          
      # Hundreds of lines here if needed

environments:
  production:
  staging:
  testing:

I do understand that this is a pretty fundamental and breaking change that I'm suggesting, however I think it provides an improved workflow with less duplication.

From the parent directory I can do

stack_master apply production
stack_master apply staging
# etc

Anything I'm missing?

[nitehawk]

@johnf That POC concept makes a lot of sense to me overall. How difficult would it be to add something to the environment to use different aws cli profiles? While there are a variety of mechanisms for credential management, supporting cli profiles would enable multiple account usage in a fairly transparent manner.

Multi-region deployments would be easy enough to setup with this design as additional environments as well.

[johnf]

I just added support for profiles in 6627f5a

[johnf]

We might need to come up with some sort of extra syntax if we want the stack_ouput resolver to support this too.

Currently it is

vpc_id:
  stack_output: [region:]stack-name/(OutputName|output_name)

Perhaps we should be more explicit?

vpc_id:
  stack_output:
    region: us-east-1
    profile: account_b
    stack_name: foo
    output_name: bar

[patrobinson]

I wouldn't try and add too much to your PR, getting stack outputs to work across accounts isn't really relevant

[johnf]

I'd love some guidance here from the maintainers on their thoughts on this POC.

The main change I'm proposing is a breaking one, so if there is interest then we could start thinking about a 1.x roadmap create a branch etc and start adding these types of things in there.

I'm happy to help out and start breaking this up into appropriate PRs etc but there isn't much point unless the maintainers and other community members are on board.

[johnf]

I wouldn't try and add too much to your PR, getting stack outputs to work across accounts isn't really relevant

Agree. Just dumping stuff in here for now as a POC, needs to be broken up if it moves forwards

[johnf]

Any more interest in this?

[devt]

I think what johnf proposes makes a lot of sense. I also have use cases to deploy the same set/list of stacks each maintained as a separate CF template to a number of different accounts. These VPCs in these different account might or might not be in the same region. We are using 3 regions so far. So I like the approach of defining 'environments' which are a set/list of stacks if I understood this correctly

Right now I have to list the exact same set of stacks in each possible region just because some accounts use one region and others another.

region_aliases:
  us: us-east-1
  eu: eu-west-1
  gov: us-gov-west-1
  
stacks:
  us:
  
## new managed policy for Service Catalog users: cldauto, cldmgmt & cldarch depend on it
    cataloguser-policy:
      template: cataloguser-policy.cf.yml

## SVC role for System Manager Service.  cldauto depend on it
    cldauto-ssm-svc:
          template: cldauto-ssm-svc.cf.yml
          
    cldauto-fed:
      template: cldauto-federated-role.cf.yml
    cldmgmt-fed:
      template: cldmgmt-federated-role.cf.yml
    cldarch-fed:
      template: cldarch-federated-role.cf.yml
    cldwebops-fed:
      template: cldwebops-federated-role.cf.yml

   eu:
    cataloguser-policy:
      template: cataloguser-policy.cf.yml
    cldauto-ssm-svc:
          template: cldauto-ssm-svc.cf.yml
    cldauto-fed:
      template: cldauto-federated-role.cf.yml
    cldmgmt-fed:
      template: cldmgmt-federated-role.cf.yml
    cldarch-fed:
      template: cldarch-federated-role.cf.yml
    cldwebops-fed:
      template: cldwebops-federated-role.cf.yml      

  gov:
    cataloguser-policy:
      template: cataloguser-policy.cf.yml
    cldauto-ssm-svc:
          template: cldauto-ssm-svc.cf.yml
    cldauto-fed:
      template: cldauto-federated-role.cf.yml
    cldmgmt-fed:
      template: cldmgmt-federated-role.cf.yml
    cldarch-fed:
      template: cldarch-federated-role.cf.yml
    cldwebops-fed:
      template: cldwebops-federated-role.cf.yml

In this particular case I do not need to be passing different parameters to the stacks, but if I did they would have to be separated by 'account', not by region.