Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery Meetup

Editor's Notes

• #2: 182 slides for last audit 85 slides with a lot of content for Gene Kim - Phoenix Project
• #3: The last two bullets are key - COMPENSATING CONTROLS IF IT IS HARD DO IT MORE OFTEN
• #5: – Super detailed including RACI charts, tractability matrices…even included who to invite to meetings and required meeting invitations to be saved! Result: Development came to a grinding halt. – A second SDLC was written which stripped away 90% of the full SDLC rigor Result: Every project became SDLC Lite. Agile – We obviously blamed Waterfall for our shortcomings (it couldn’t be us). We went Agile….sort of. Result: Chaos. Team was not ready for quick sprints, documentation wasn’t done, code wasn’t finished….. Agile & Continuous Delivery – A proper implementation of Agile with the technical craftsmanship that is required. Result: A successful and strong base in which to build
• #10: Counter intuitive - You may not get the best bang for the buck
• #11: David Farley answers posts in LinkedIn
• #12: Some auditors and examiners are very familiar with Agile. Many even have CSM and CPO certifications. However, some are entrenched in Waterfall. Also, keep in mind that many guidelines that examiners are required to follow are based on the Waterfall methodology. Shows that the check points still exist, just a little in a different order or a little more often
• #13: Infrastructure as Code enabled us to reduce our OS patching frequency from quarterly to every two weeks A finding from our penetration testing exercise is added to the next sprint which means it is in production in just over two weeks A change in our process is added and adopted by the feature team by redefining the definition of “done”
• #14: WHAT IT ALL COMES DOWN TO IS MANAGING RISK A quarterly release cycle contains months and months of code. This is harder to test, harder to perform a proper code review, and significant amount of change to the application is introduced at one time. Changes performed in small batches reduces the risk of any one release. Even if the entire release needs to be backed out, only two weeks of work is lost which reduces the company’s financial risk Schedule risk can also be mitigated by reducing feature creep, gold-plating and by keeping stakeholders aware of progress
• #15: At this point, you may have shown some good insights about Agile, but chances are, your auditor already knew about it. CD is a different story. There are LOTS of levels of maturity
• #16: Going back to the Bible We regularly found ourselves falling behind in signing off on documents even though the work was done and already in production. With automated processes every step can be tracked so the need to manually say it was done is no longer needed. Log files and reports can show that all the appropriate work was completed.
• #17: Auditors and Examiners regularly want to understand how environments are kept in sync. This is complicated for endless reasons like history, culture, funding, etc. It becomes exponentially more complicated if you have multiple development environments, multiple QA environments, etc. However, a CD Pipeline which performs the exact same deployment in every environment forces a team to have consistent environments.
• #18: Infrastructure as code will overwrite any non-standard changes. Changes need to be made in Development and then they are migrated through the environments. When you get to the point where your entire server is managed by a configuration management tool like Puppet or Chef, it will mean that the server will be rebuilt to the standard with every release.
• #20: MANAGEMENT OVETSIGHT It is important to note that a little while back we changed our rule set which resulted in more critical issue Static Code Analysis Tools brings constant awareness of code quality to management. This is a fantastic tool to demonstrate maturity and constant improvement. We use Sonar, but the key is to have a tool in place and to take action on the results.
• #24: RESULTED IN A SURPRISE Automated tests are key to any Continuous Delivery Pipeline otherwise it is simply not possible to have regular deployments due to the time needed for manual regression testing. However, test automation is also very helpful for auditors and examiners to see because it helps explain how code can be deployed faster but the team still has the same (or greater) level of confidence It is important to show that your pipeline will STOP until any failed automated test is corrected.
• #26: Another important tool in any Continuous Delivery Pipeline is a Repository Management Tool like Nexus. This tool is used to hold binary files for deployments as well as all of the dependencies that the application and infrastructure pipelines may need.
• #28: Probably one of the more unpopular changes is to switch to an electronic board like Jira. Many teams are very fond of have note cards and post-its on walls, but digital boards are more auditable. Once you make the switch, you will have lots of unexpected benefits….here are some more!
• #29: We use a Jira plugin called Group Sign-Off for Jira. It allows a story to capture key sign-offs from management, security, and compliance. We include a sign-off story in every sprint and now no longer need to print and get manual signatures. Using permissions in Jira, I can only sign-off as myself.
• #30: This was one of those dreams that I never thought would come true, but with the Xporter Plugin for Jira we are able to perform a mail merge into our SDLC template. We capture the stories and sign-offs for every release to fulfill auditor and examiners requests for documentation. We do still have the manual step of maintaining master requirements documents for major pieces of functionality.
• #32: Using the logs captured by Jenkins, we created a report to show when each step in the pipeline occurred and who initiated it. The report is the last step in our Pipeline. It emails a copy of the report to all interested parties and places a copy in Nexus for archival purposes.
• #33: All of the activity logging is beneficial as it shows that all the steps were performed and who performed them. However, they are also rich with information about your own processes. Graphing the metrics from various points in your development process will again show that management is involved in the process and providing proper oversight.
• #34: With a self managing team that is making small changes every two weeks, it is easy for management to not know exactly what is going on. We added a Sprint Planning Review Meeting after every sprint planning session. This was facilitated by our ScrumMaster and was found to be very helpful to bring Security, Compliance and anyone else into the process. Not only was this meeting helpful, but it also demonstrated to auditors and examiners that we were thoughtful about the process and made improvements when needed.
• #36: ACTIVE DIRECTORY - NEED TO HAVE A PERIODIC REVIEW OF WHO IS IN GROUPS Our QA Team performs and important check to ensure that a quality product is deployed. They are also a critical team to maintaining separation of duties. They don’t write the code and they also have the best understanding of how the applications should work. We found they were best suited to facilitate the deployment. They control:
• #37: We have four Operations Teams: System Engineers, Application Engineers, Network Engineers and Database Administrators. The System and Application Engineering teams are very interested in deployments so they can be extra vigilant in their monitoring during and after deployments. To keep them involved we added steps in Jenkins to allow:
• #38: The rapid deployments still resulted in the occasional incident where someone was caught off guard. We addressed this by adding an email step before Staging and before Production to notify all interested parties that a deployment was coming through.
• #41: Only a select number of senior developers and architects have the ability to merge code into the development branch
• #42: Users ended up with Local Admin and Regular User Local Admin users are only allowed to MERGE - by policy Wrote custom code to only allow ACTIVE DIRECTORY USERS to check in and approve PULL REQUESTS
• #44: The number one concern that an Auditor will raise is how can they be sure that all the wonderful tools that we are using are properly managed. One examiner quickly realized that an administrator of Jenkins would have tremendous power in our environment
• #45: AGILE TEAMS NEED TO POST THEIR DEFINITIONS OF DONE IN CONFLUENCE (WIKI) REGULARLY AUDIT USER LISTS - ACTIVE DIRECTORY - In a recent exam, I was very impressed how quickly the examiner recognized the power of some of the tools like GIT, Stash, Jenkins and Puppet. He immediately wanted to know how we ensure that the appropriate permissions are in place. Also, think through how you will manage the environment of CD tools. Typically, one doesn’t have a migration path for making changes to Jenkins, Sonar, etc.