Using AWS to Achieve Both Autonomy and Governance at 3M

Editor's Notes

• #6: Been in business for 30+ years Develop products and services that help our customers produce accurate documentation and medical coding to improve quality of care and reduce cost.
• #7: The US is moving from a fee-for-service based medical care to big data driven population health Measuring performance and effectiveness of care Determining actions to take on that performance for improvement
• #8: 24 states have our adopted our systems 87 % of the population is covered by our systems 1% of the Gross Domestic Product is being risk adjusted with 3M HIS methodologies (products and services) Lots of records Lots of dollars
• #10: Bottleneck=The amount of time it takes to do the action and waiting on the availability of the team. How long it took to get to production on some of our deployments
• #11: Get software into the hands of customers as fast as possible.
• #12: Rob Brigham
• #13: Building the CI/CD platform team Choosing the right technology Security Find the right service Find a hungry team Embed with the team Establishing a Feedback loop
• #14: Needed a balance of engineer types and consulting engineer types If you don’t have it in house, bring in consultants and rotate FTEs into the team
• #15: Don't try to over engineer to solve all types of delivery Don’t try to figure out what you are going to need, figure out what you do need Know that your CI/CD platform is iterative, like any product it will get better over time Use native AWS services/Lambda/SaaS over instance-based infrastructure when possible
• #16: Security involved in the cu CI/CD needs to have security baked into the process Start building the platform with the Security team to gain buy-in early Help the security team become consumers of the platform so they can be champions Regulated Data Development with Guardrails Sensitive Data requires unique control frameworks that must be implemented.
• #17: Find an easy to deploy service Small, stateless, a web app? Get that thing to production don’t worry about containers, microservices, just yet. Keep trying new services, wait for patterns to develop, iterate
• #18: Find an easy to deploy service Find a team that is eager Some teams want in just because it’s the hot new thing http://lghttp.32478.nexcesscdn.net/80E972/organiclifestylemagazine/wp-content/uploads/2015/02/Hungry.jpg
• #19: Explain the teams on the graphs Explain the bullet points Onboarding team works closely with the Automation Engineering Team Communicates App Team challenges to Automation Engineering Team Acts as champion for App Team issues to make sure they are captured for future Teams and Pipeline Factory enhancements Hands the steering wheel when app team is ready
• #21: Consistent CICD Pipelines and process at scale
• #22: James covered challenges and the approach to addressing 3 parts to solution Pipelines – every commit can make its way to production with minimal human intervention  (SPEED/AGILITY) Self service – teams can create and manage their own pipelines  (AUTONOMY) Monitor – guardrails to keep people from hurting themselves  (REPUTATION & COMPLIANCE)
• #24: Restate problem – manual handoff Some automation, but still requires support from a centralized team
• #25: Use CodePipeline for automating deployment workflow ### All deployments must be done via pipeline ### Triggered by commit Single pipeline per deployable application/service ### Only yes/no input All infrastructure defined as CFN by developer ### Everything in code
• #26: Stages… Actions… Integrations (CodeCommit, CodeBuild, CodeDeploy, Jenkins) !!!UPDATE notes
• #27: 2 repos – one for app, one for IaC…allows separate of roles inside a team Pipeline is trigged when either one changes
• #28: ### Define all Jenkins jobs as JobDSL in the IaC repository Every pipeline execution runs the DSL
• #29: Source is built, unit tested and packaged We’ll come back to CfnNag later….
• #30: 3 stages…one per environment (automated testing, manual testing, production) ### Only manual step is between each env...approve/reject
• #31: Launch infrastructure via CloudFormation templates defined in the IaC repo ASGs, ELBs, DBs
• #32: Deploy app that was built previously to new infrastructure
• #33: * Run end to end tests…selenium, resteasy, postman/newman
• #34: Blue green switch at the ELB to the new ASG !!!UPDATE – blue/green
• #36: New problem…how to allow self-service to provision pipelines? Don’t want to allow folks to create manually Needed a pipeline factory!
• #38: Least privilege - Control who can create pipelines via IAM. Govern – Pipeline is creating exactly as intended as users can only create whats in the approved template. Versioned - Changes can be versioned allowing users to consume changes to pipelines at their own pace Declarative > Imperative - Easier to manage as CloudFormation does a great job of converging incremental changes. Simply declare the desired state of your resources and CFN will make it happen…rather than you having to write the code to do that hard stuff ### CloudFormation is king – easier to version and apply incremental changes ### CloudFormation service role – a role that only is used by service catalog/cloudformation that has all the access…can’t be assumed by users ### ServiceCatalog to provide self service with governance
• #39: Demo script…(to be recorded) Create team via SC Login to Jenkins View list of created stacks (cross account) Create pipeline via SC View CodePipeline View Jenkins View CodeCommit Watch pipeline succeed
• #40: Service Catalog creates top level stack Custom resource backed by lambda function, creates nested stacks in other accounts using IAM role Can reattach to existing stack, useful for KMS keys and S3 buckets ### Retain important resources – buckets, keys, databases
• #41: Custom resource One per account, uses AssumeRole to jump accounts Shared template for all accounts, versioned DeletionPolicy…retained and reattached
• #42: Self service documentation How to get started How to solve common problems Changelog and migration documentation
• #44: Teams create the CFN for their ELBs, ASGs, Route53, RDS The cfn-nag tool looks for patterns in CloudFormation templates that may indicate insecure infrastructure. ELBs that are open to outside Security group rules that are too permissive (wildcards) Access logs that aren't enabled Encryption that isn't enabled ### static analysis before deployment
• #45: Rules defined via custodian DSL Deployed as lambda functions Perform notification and remediation
• #46: Look for public buckets Automatically remove grants and website hosting Notify the resource owner ### setup processes to assess and enforce policy compliance !!! Mode/type
• #47: Look for instance missing ”Cost Center” or “Team” tags Stop the instance Notify resource owner
• #48: Teams can define their own tests (functional or non-functional) as lambda functions Modify S3 bucket ACL -> failed build IAM role trust policy with non-HIS account -> failed build Permissive security groups !!! UPDATE - icon Dynamic testing framework for infrastructure and application level functional and non functional tests Verify Infrastructure aligns with AWS Best Practices (AWS Security Epics) and your own organizational governance Application Level Functional Tests (Call my endpoints and assert the response) Non-Functional Tests (Terminate instances in auto-scaling group, verify resiliency ) Framework allows for dynamically testing AWS best practices like (AWS Security Epics) Framework capable of running cross account tests, in multiple accounts Security Tests (Organizational / BU Level) are run in SecOps, but test infra in other accounts Application Tests (Product Level) Created by the app team are executed in the deployment account(s) Framework that can be directly integrated with the pipeline or used independently with minor changes Embraces DevSecOps allowing the security team and the application teams to build security into the development process Organization Level Test – Test define to verify enterprise or business unit requirements Product Level Test – Test written by the product team to verify security, functional, and non-functional requirements
• #49: Single CW dashboard showing metrics for each pipeline SuccessCount FailureCount CycleTime RedTime GreenTime ### monitor health of pipelines !!!UPDATE – new picture
• #50: Triggered by each CW event Recorded as CW metric, pipeline/stage/action as dimensions Dashboard, built nightly via lambda that queries CW metrics !!!UPDATE - typo
• #51: SAM Defines both the function and the event rule
• #52: SAM Runs nightly !!!UPDATE - cron schedule
• #53: Continuous Delivery ### Everything in code ### Deployed via pipeline ### Triggered by commit ### Only manual step is between each env...approve/reject Self Service ### ServiceCatalog to provide self service with governance !!! UPDATE – add bullet point
• #54: Self Service ### CloudFormation is king – easier to version and apply incremental changes ### CloudFormation service role – a role that only is used by service catalog/cloudformation that has all the access…can’t be assumed by users ### ServiceCatalog to provide self service with governance ### Retain important resources – buckets, keys, databases Monitor ### static analysis before deployment ### setup process as guardrails that assess and enforce policy compliance ### monitor pipeline health  !!! UPDATE – add bullet point