Be in any game long enough and you start to notice patterns. The pendulum begins to swing in different or new directions altogether. This is especially true in technology. I haven’t been in AWS all that long and I already can see areas where things begin to drift.
This time – Multi Organization configurations
It is hard to explain the feeling you get when you just know something is about to happen or is already in the works. I could be way off on this. To those already struggling to maintain their AWS Architectures, I may look like a lunatic. Instead, I hope I lay out a new idea and ways to consider building out cloud architectures.
I also want to get this thought down and public prior to AWS re:Invent 2022. Just in case things really start moving in this direction, I can have a little proof that I knew what I was talking about.
I have no inside information. This is all based on assumptions and derived from my opinions about some of the most recent service and service feature announcements.
If you are paying attention and care enough about a certain area you begin to see patterns emerge. For me, that is AWS management and governance. My career in AWS has revolved almost entirely around the management of environments. So anytime a new service announcement relates to managing AWS Accounts as a whole, I take notice.
What I’m seeing
The delegation of AWS services to a central account has proliferated. This has a tremendous impact on securing critical AWS accounts.
Protecting the core (Management) account of the organization from inadvertent access or access by excessive individuals is critical. The management account really is the key to the kingdom. Getting administrators out of that account is a huge step.
The next step isn’t a big leap. Architects can already delegate main services administered for any entire AWS Organization to a single account. In the case of CloudFormaiton StackSets, you can delegate up to 5 accounts. This process leverages the AWS Organization and service role trusts. I don’t know all the deep interworking of those service trusts.
I can make an educated guess, that it wouldn’t take too much effort to elevate that trust one more degree higher to accounts outside of an Organization to accounts that are members of a different AWS Organization.
An Organization of delegated administrator accounts
Imagine a new AWS Organization that isn’t configured like your current organizations with workloads, Control Tower, Service Control Policies, and shared networking.
Image this organization having central control across multiple other organizations for the following:
-
Access Control (i.e Identity Center, formerly SSO)
-
Service Catalog
-
CloudFormation StackSets
-
Security Hub
-
Inspector
-
SSM
-
Billing
-
etc.
Just as you can invite accounts into an AWS Organization, at some point I believe AWS will enable the ability to invite delegated service administrator accounts to manage other organizations.
You may or may not have to first delegate to a central account INSIDE of an organization. I think you could and should. Administrators and security teams can have a view of their landscape from inside the individual organization. Global admins could see everything from the `Org of Orgs.`
Break Glass Access Control
AWS should have an external service that can be protected with MFA, like AWS QuickSight or Amplify, for example, that has its own external login portals outside of the AWS Console. This service could be used to manage break glass access to any account through the use of assumable roles.
Until there is an external service like this, there isn’t a great way to set up secure emergency access to accounts in an organization.
The most basic configuration of a multi-org strategy you could start with today is setting up a new organization just for external break glass access.
This model of access would be an Identity Center configuration in a region separate from the deployed regions of your other organization(s). In some cases, but not all, this could prevent access issues when there are problems in the region where you have your Identity Center configuration deployed. Since almost all authentication services run through us-east-1 anyway, your results on this one may vary. At the very least you have a separate AWS SSO/Identity Center solution in the event something bad happens with your External IDP. For instance Azure AD or Okta.
How do admins gain access to any accounts if your External IDP is down or broken?
Out of the box, this is likely the `OrganizationAccountAccessRole` from your Management account. But you shouldn’t use your Management account for break glass access. This likely means you have root user access or IAM users for your Management account. You shouldn’t use the Management account for much of anything really.
The best option is, in your existing workload Organization(s) you have a StackSet that deploys an assumable role to all accounts. This role is then accessed from an account or accounts in your new organization used just for break glass access.
You can have a specific permission set dedicated to only being able to assume a specific role in all, or even some, of your critical accounts across your other organizations.
Break glass access from a special account isn’t a novel idea. We’ve all needed it at some point. While this post about having an AWS account just for getting into other AWS accounts doesn’t meet exactly what I am talking about, it gives you the right idea.
Workload Access control
Think about the number of people (developers, administrators, security professionals, newly hired, novices, etc.) that will be accessing your single AWS Organization. Now think about how easy it is for someone to be improperly added to the wrong Identity Center Group or individually added to a permission set on a very critical workload account.
If you are using SAML or some other solution for access, the problems compound with the more accounts you have. There aren’t many solutions that allow the nesting of groups of users that port nicely into access within AWS.
Having multiple organizations allows you to have multiple Identity Center configurations and therefore multiple External IDP configurations. You can have dedicated access to your organization where users simply don’t have access to a specific organization because their workload accounts don’t exist there.
Billing & Cost Allocation
In any organization, there are two types of development.
There is the development directly related to service and/or feature enhancements of production products. This type of work supports build releases and bug fixes of revenue-generating solutions.
Then there is the R&D or dinking around your developer teams conduct to test and learn a new feature or service. Here services may get left on or accidentally introduce vulnerabilities into your sanctioned product development accounts.
A solution…
Give all of your developers an account in a Sandbox/R&D Organization. Lock it down to one region. Give every account a monthly budget and alarm and let your developers tinker in there. This way R&D cost is separate (and consolidated) inherently from development that is tied directly to your production product or service. No additional tags or access restrictions are required. Easily terminate accounts or nuke services in R&D accounts.
All of which can be automated nightly as well. Automated without the concern of hitting anything in production or critical development.
You might also consider creating an organization just for suspending accounts or transferring ephemeral accounts.
Product, Service, or Department Segmentation
With access control sort of figured out, suddenly managing multiple organizations doesn’t seem so icky. Bucketing your accounts into different AWS Organizations instead of just Organizational Units has a lot of benefits as you scale out your architecture.
I don’t personally know your organization’s departments or structures. However, there are many places where this might work for you.
One example is anywhere your company might sell, cut ties, or have services that can operate on their own without global governance or management being tied to another group of services.
Do you have a business unit that operates self-sufficiently and has its own cost structure? Even if you need to have shared networking you can have these solutions in their own organizations. With a dedicated invoice, credits, and access controls.
Could a whole arm of your organization undergo acquisition by another company? Maybe that should be in its own AWS Organization.
Are you in the business of acquiring part of or other businesses entirely? How can you keep those accounts in a different organization indefinitely or at least until you have a good consolidation strategy?
The answer to all of these could very well be unique AWS Organizations for each situation.
Compliance
Security Hub checks, workload security, and compliance frameworks can be managed pretty easily across a single organization. But what happens when you have multiple subsets of accounts that all require an exemption to some framework or benchmark standard?
Wouldn’t it be nice to look at a single pane of glass that is all in one compliance context?
If you have dozens or hundreds of accounts that fall under the same standard(s), anomalies stand out. When all of your Security Hub checks or Inspector results line up across all accounts of the same compliance standard, you have a true understanding of your security posture.
When you have to account for the small number of accounts that are allowed to have EC2 instances directly accessible from the internet, those results distract you and can catch your eye every time you are reviewing your dashboard.
This makes you more susceptible to missing a true configuration issue when it pops up. Since you are already expecting to see red.
Where would this setup be helpful?
Don’t get me wrong. There are caveats and nuances to all of this.
Especially concerning billing. Any credits you may get from AWS, things like Reserved Instances (RIs), Savings Plans (SPs), and Enterprise Support are all factors. I have solutions for some of that today. For instance, creating accounts just for the purchase of RIs or SPs that can be moved from one organization to another if needed.
I’m certain this will be taken care of by AWS as this all evolves. Again. I could be completely wrong about all of it.
A relatable example
I get it. This all seems like it could be considerably more complicated than it is worth. You are not wrong. The initial setup takes a little bit of extra work.
Once the initial foundation is laid and automation is in place, this all just makes a lot more sense. This solution may resonate with some of you.
The solution I have recently been deploying is as follows:
-
A management organization
-
primed and ready for the central delegation of services I mentioned above
-
central S3 bucket(s) with permissions for all other organizations to access global packages and CloudFormation Templates
-
Identity Center with dedicated permissions for assuming all other accounts across all other organizations
-
-
Networking – something I will get into deeper in a future post but think…
-
Direct connect terminated in networking account
-
Transit gateway connections to a Shared VPC in a networking account in each organization
-
VPC shared subnets to all accounts within the organization using RAM in the Organization
-
-
automation account
-
a single entry point with an IAM trust to automate across N other organizations
-
this prevents unique credentials in automation for each organization.
-
a single set of credentials for one account that can run automatic assumptions across all other organizations.
-
-
Workload Organization 1
-
a specific product/service
-
a very specific team of developers that needs access
-
-
Workload organization N
-
any number of additional organizations
-
no cross-team access is required – different dedicated developers
-
-
Back office services, administration, and automation
-
Finance services
-
HR
-
etc
-
All of this is being set up with automation in place using GitHub Actions or GitLab-CI, I deploy identical resources to all accounts across all organizations. I can do so with the same number of operations as using a single Organization. It’s just a small tweak to the automation.
At a larger scale
I have over a decade of experience in the federal system. I am going to use that to expand this entire idea to a much larger scale. This way you can see where this might apply anywhere from a smaller local business with a heavy technical workload to a much larger enterprise scale.
The government has many ‘Departments’. Any time you see a department, that means there are smaller bureaus or branches within the department.
For instance, the Department of Defence (DOD) has the Army, Marines, Air Force, Navy, Coast Guard, and Space Force.
The Department of Interior (DOI) is a whole hodgepodge of bureaus that honestly didn’t fit very well anywhere else.
The model of multi-organization I discuss here would work great for this style of hierarchy. Anywhere central governance and compliance all matter and looks a specific way. Everything rolls down from the top including funding, but yet each individual branch or sub-division operates very differently.
At the top, there is global administration and compliance. Each with their core services, monitoring, logging, etc. delegated and transferred to the mothership AWS Organization.
Inside each organization are the unique access requirements for each sub-division. Since they all have their own Enterprise Directory systems anyway, they can manage their own Identity Centers. The teams of people doing the work within their organization manage the compliance up and down the chain.
This is probably the most extreme of the use cases but it helps bookend the unlimited options in between.
Wrapping up
Finally, the driving factor behind all of these options is enablement.
Why did I even start to think this way in the first place? I asked myself these questions:
-
How can you get the biggest bang for your buck from a management standpoint where you don’t have to constantly be correcting mismanaged resources, or mistagged resources?
-
What is the least common denominator where a set of accounts belong together?
-
Where can I reduce risk and increase autonomy by letting the platform do my job for me?
Breaking out account types and/or products into their own Organizations seemed like the most logical solution. However, I have over the last to years focused on deploying many new AWS Organizations. All with Control Tower, Customizations, and more recently Account Factory for Terraform.
What are your thoughts on all of this?
Where do you see gaps or concerns in my architecture?
I would love to hear from you.
If you enjoyed this post or would like to learn more about how I look at AWS architectures, Management, and Governance please subscribe to my weekly newsletter – https://unlimitedleave.com.
