When Management Asks You: “Do You Accept Agile as Your Lord and Savior?” - CircleCityCon 10 - 2023
- 1. Dan Lagos “AdmFord" - CircleCityCon 10 - 2023 When Management Asks You: “Do You Accept Agile as Your Lord and Savior?” Agile In Operations, Can It Actually Be Done?
- 2. What Is Agile In The First Place? The Basics • Welcome change, even late in a project • Iterate work frequently, anything between a month to a week. • Business people and project engineers work together regularly. • High trust environment, you accept that your team knows what they're doing • There is adequate experience in the team, and adequate documentation for training • The most e ffi cient and e ff ective method of conveying information to and within a project team is face-to- face conversation (either in person, or via video/voice chat) • A sustainable pace is maintained throughout the project. • Simplicity is key, don't do work that doesn't need to be done to complete the project. (Think: Minimal Viable Product) • The best architectures, requirements, and designs emerge from self-organizing teams • With every iteration, re fl ect on how to become more e ff ective, tune and adjust behavior accordingly
- 3. Origins of Agile From The Factory to the O ffi ce Floor 1. Identify the constraints (bottlenecks) 2. Decide how to exploit the constraints (quick improvements using existing resources) 3. Subordinate (Review the process to make sure the constraint is actually needed) 4. Elevate the system’s constraints (constraint still there, consider further actions or investments needed to eliminate it) 5. If the constraint breaks (is eliminated), go back to step 1. But don’t allow inertia to cause a new constraint • TOC advocates keeping inventory of products low. • You don’t make new inventory when you can’t sell it • You keep in-work inventory low by using small batches • Time to process an element of inventory is key to fi nding where bottlenecks appear • Inventory is anything that requires labor to create, or modify (for us, that’s tickets & tasks) Theory of Constraints Any system is limited from achieving more of its goals by a small number of constraints, and by focusing to identify and restructure the organization around them, we limit their impact.
- 4. Origins of Agile The Toyota Way • Continuous Improvement • Long Term vision for a company or a team • A reasonable, but in reality, potentially unachievable goal • Example: “The team will respond to any new IOC/IOA/Vulnerability and implement an alert in production or remediate it within an hour, while following standard operating procedure" • Continuous Improvement (kaizen) • Find the source of a problem to make good decisions • Respect for people • Respect others, and take responsibility to do our best to build mutual trust in a team • Stimulate personal and professional growth, share opportunities of development and maximize individual and team performance
- 5. Origins of Agile The Toyota Way (part 2) • Right Process • Create a process that can bring problems to the surface • Use a pull system to avoid overproduction • Level out the workload (don’t overload workers) • Build a culture where it’s ok to stop a process in order to fi x it, when problems are identi fi ed • Standardize tasks and processes • Use visual control so no problems are hidden (dashboards!) • Use already tested technology (don’t reinvent the wheel) • Add value to the organization • Grow leaders who understand the work, live the CI philosophy, teach it to others • Develop people and teams who follow the CI philosophy • Work with partners & suppliers by challenging and helping them to improve • Continuously solve core problems drives organizational learning • Go see for yourself the situation to understand it • Make decisions by consensus, considering all options, then implementing the solution quickly • Become a learning organization through re fl ection and continuous learning
- 6. It’s Basically Project Management But Without The Project Managers (To A Point) • Every plan fails when confronted with the enemy • The further you plan out work, the more of a chance unexpected work or circumstances will wreck your schedule • Waterfall works when plans don’t change • Increasing fl exibility means breaking work up into smaller more manageable chunks • Iteration based on the chunk size, work with team leads, management, and employees to identify issues and solutions quickly.
- 7. Framework Letter Jumble SCRUM, KANBAN, SCRUMBAN, etc… SCRUM SCRUMBAN KANBAN Team Members Recommended members between 3-9 No Speci fi c Limitations on the number of team members No speci fi c limitation on the number of team members Team Roles Members are assigned di ff erent roles & responsibilities No Roles Members are generalists or specialists Work Cycles Sprints that can last from 1 to 4 weeks 2-Week Iterations with continuity (the board is not cleared) Continuous Work fl ow Rules Follows Strict Rules Finds the middle grounds between scrum & kanban with moderate rules Relaxed and Flexible Task Assignments Assigned to the team members Members choose their tasks Members choose their fast- paced (smaller) tasks Limits Based on the current sprint Limits placed on the work-in- progress Limits placed on the work-in- progress
- 8. Scrum Meetings The Structured Work Process (That can, or usually be, a PITA) Frequency Attendees Time Needed Objectives Sprint Lasts anywhere from 1 to 4 weeks Sprint Planning First day of sprint Product Owner, Scrum Master, Scrum Team Potentially an hour, depending on team size Set a sprint goal Daily Scrum Daily Scrum Master & Scrum Team No more than 15 minutes What did you do yesterday? What are you doing today? Any Blockers/Issues? Sprint Review At the end of a sprint Scrum Team, Product Owner, Scrum Master 2 - 4 hours Demo of work done user stories - con fi rm and decide on incomplete ones Assesement of Backlog Sprint Retrospective Between Review & Planning meetings Scrum Team, Scrum Master, Product Owner An hour What was done well What didn’t go as planned Improvements for next sprint Backlog Re fi nement Every other week depending on the duration of the sprint Scrum Master, Product Owner, Potentially Scrum Team No de fi ned duration Prioritize backlog items Alight backlog to KPIs Appropriate sizing of backlog items Add more detail
- 9. Frameworks and Team Evolution Using the Stages of Team Development to do Agile Scrum A structured work environment provides bene fi ts Scrumban Easing of structure Productivity is stabilizing Kanban Less structure needed More autonomous work Forming Storming Norming Performing Change in Composition • Frameworks, while not necessary to fully implement, can be used to help in team evolution.
- 10. Agile Is Not Simply Implementing a Framework Stay Away From “Cargo Cult Agile” • An Organization that wants to implement “Agile" can’t just order the use of Scrum or Kanban across the board. • Every operations team will have di ff erent organizational, and business requirements • Frameworks can be implemented partially, or in a hybrid manner (SCRUMBAN) • Implementation is a two way street • Organizations must change to give Project Managers / Product Owners / Team Leads more liberty to initiate & lead changes • Team members have to speak up when they identify problems or constraints
- 11. Actually Putting Agile to work, DevOps The Culmination of Agile in the Development World • In development, the end state of Agile can be said to be DevOps. • DevOps has the same principles as Agile, while not necessarily following strict frameworks. • It de fi nes objectives regarding the types of work and techniques to achieve them • It expects good knowledge of what the team members are doing, and creating/implement tools that they can implement changes frequently. • We see this in things like the Software Development Lifecycle (SDLC)
- 12. Types of Work and Priorities The Four Types of work Gene Kim in the Phoenix Project and DevOps handbook de fi ned the four types of work in a business • Business Projects (Highest Priority) • Business Initiatives, most of development and engineering work • Internal IT Projects (2nd Highest Priority) • Infrastructure and IT Operations • Updates and Changes (Normal Priority) • Often Generated from the two previous types of work • Unplanned Work or Recovery Work (should be limited as much as possible) • Incidents and problems generated by other work
- 13. The Three Ways of DevOps How CI/CD Works 1. Systems Thinking/Flow of Work • All work should ideally fl ow in one direction, such as across a KANBAN board (from new, to in progress, in review, to done) • Any fl ow back in the process means that there are unidenti fi ed potential issues or constraints that need to be addressed 2. Amplify Feedback Loops • Learn from current processes and fi nd improvements • Implement improvements to the processes 3. Experimentation • Try multiple ways of improving processes and work and test which work better • In development, the best example of this is A/B testing
- 14. DevOps foundations in Security (DevSecOps?) Enterprise vs MSSP • Enterprise & Team Projects take priority • Maintenance and Operations work (tasks for other teams, installation of software, etc) fall within the Team Projects priority • Incidents take engineers away from improvements and should be limited • 80% Project & Ops work 20% Incident work • Automate Incidents and Operations to meet KPIs and team SLAs • MSSP Improvements, Maintenance & Operations work may be handled by a separate team compared to Security Monitoring • Incidents are the main work done • 80% Incident work 20% identifying improvements • Based on contracts with other organizations, apply automation and process improvements to meet KPIs and SLAs
- 15. Starting the Change to Agile People, Process, Tools • Change has to come from both the top (Senior Management & Management) and within the teams themselves • Identify all the processes that your team is involved with. • Tasks received from other teams. • Incidents your team have de fi ned and how to resolve them. • Common operations work that is done frequently • Playbooks are your friend. • Word, Sharepoint, OneNote or Wiki are fi ne. • Playbooks within your ticketing system (such as the SecOps module in ServiceNow) are preferred and force analysts to follow the work fl ow, ensuring repeatability. • Creates the foundations for automating the Incident Response process. • SOAR to accelerate. • Automation is needed to increase the throughput of a team by freeing up time for other work & tickets
- 16. Change Mangement Many Di ff erent Models Available Change Management Models Process Focused People Focused Kotter’s 8 Step Change Model Lewin’s Change Model Deming Cycle | PDCA McKinsey 7S Model ADKAR Model of Change Nudge Theory Satir Change Model Bridges’ Transition Model The Kübler-Ross Change Curve Maurer 3 Levels of Resistance & Change Model
- 17. From Process to Pipeline Identify the Work process, Find Improvements, Rinse and Repeat Knowing your team’s work processes is foundational. Since it is literally understanding what work is done, and how it is done. 1. Interactive Playbooks are a fi rst step to standardize work 2. Identify dependencies (where other teams are needed) and constraints (certain tasks always falling to a single team member) 3. Implement improvements A. Automate parts of the playbooks B. Request that other teams automate some of their work that’s related to your own team’s C. O ffl oad excessive work from a ff ected team members 4. If IR to alerts has been fully automated, create new alerts based on frequency of the automated ones, to fi nd discrepancies to normal daily fl ow (ML could be bene fi cial in this) 5. New services, products and technologies are implemented regularly, so go back to step 1
- 18. Software Development Life Cycle (SDLC) The Foundation Behind Modern Development and DevOps 1. Planning 2. Analysis 3. Design 4. Implementation 5. Testing & Integration 6. Maintenance
- 19. Software Development Lifecycle Adapting the Dev process into SecOps 1. Planning: Identify IOC/IOA that you want to create an alert for 2. Analysis: Research IOC/IOA (is it even possible in your environment?) 3. Design: Create a query to search for the IOC/IOA in your current systems (have you already seen it?) 4. Implementation: Write the actual alert logic from the initial query, and commit it to the SIEM 5. Testing & Integration: Does the alert fi re? Tickets should be assigned to whom created the alert logic. A. If successful, tune as needed B. If unsuccessful, verify why it’s not fi ring (not present, not logged, or no logs) C. Assign Purple Team member to write test article on dedicated machine to con fi rm alert will actually fi re. 6. Maintenance: Run regular “drills”, making sure that all know alerts are able to detonated at will by the team to make sure everything works correctly (think, " fi re drills”) SIEM https://learn.microsoft.com/en-us/azure/architecture/example-scenario/devops/automate-sentinel-integration DOCUMENTATION
- 20. Ticket Metrics The Subtle Art of Of Proving a Team’s Work • Number of tickets closed is a disingenuous metric. It doesn’t takes in consideration the e ff ort or the result (true positive or false positive). • Automatically assigning tickets to users randomly is good, but you can’t use the count of how many tickets are closed to measure employee performance. • Pick and pull at discretion works in not overburdening sta ff , but employees with experience can identify potentially easy tickets and horde them. • Assigning points value based on priority/criticality can help, but the frequency of the tickets and the e ff ort still isn't considered. Sta ff can potentially game the points in their favor by holding harder tickets. • Scrum o ff ers another tool, “Planning Poker”, assigning a weight for a task based on the amount or di ffi culty of work to complete it. • Weight is assigned after the tickets are closed (based on closure type, true positive or false positive)
- 21. Planning Poker Stealing From Scrum, to Make Metrics Work • Planning Poker is usually done during backlog re fi nement. • Values are decided on how much work a task takes, and NEVER how much time it would. • Humans are more visually oriented, so measuring things by estimating size or quantity can actually be more accurate. • Plus, wisdom of the crowds helps as assigning values is a team decision. • Planning Poker can use a modi fi ed fi bonacci sequence to estimate work. 0, 1, 2, 3, 5, 8, 13, 20, 40, 100 • In doing so, you do not take in consideration SLAs or KPIs • Track these separately, as adjacent goals to achieve • The law of averages in the end comes to our rescue regarding metrics • Consider refactoring values for incident types regularly (every six months or on frequency of changes to team composition)
- 22. x̄ and σ Average and Standard Deviation in a Normal Distribution • In a Normal Distribution, the average (mean), x̄, is at the central point of the curve • The standard deviation, σ, measures how dispersed the data is compared to the average (mean)
- 23. Out of Bounds Tickets When Response Times Exceed Standard Deviation (Both Positive & Negative) • When they take more time • Dependencies can be to blame, team members are waiting for other teams to complete associated work • Employee needs additional training or is a new team member and needs to improve • There potentially is a problem with a tool the team uses • These can be the fi rst targets to quickly improve SLA results by automation or process fi xes • When they take less time • An employee has found a way to automate their work • Congratulate them, and see about standardizing their procedure! • An Employee has been closing tickets by only doing the work partially. • Retraining might be required • Closer monitoring of employee’s work • Termination of employee
- 24. Track the Averages Track Team Progress • The average time it takes for a job with a certain Estimation of E ff ort (Planning Poker value), even when not linked to an KPI or SLA, can provide valuable information on the team. • As the team evolves (forming, storming, norming, performing), the average time it takes for work should drop month to month. • This includes performance improvements done with automations and improvements done in conjunction with other teams. • Time saved = Money Saved. Doing more tickets in less time means improved e ffi ciency. This allows management to o ff er a metric of money saved over time to senior leaders at an organization. • Changes to teams can raise these averages, seeing how fast they start dropping again can indicate how well the team is working together.
- 25. Useful Books Some Study Materials • Foundational Texts • "The Goal: A Process of Ongoing Improvement” by Dr. Eliahu Goldratt • “Toyota Kata” by Mike Rother • “The Phoenix Project” by Gene Kim • “The DevOps Handbook” by Gene Kim • Useful in my opinion, as Security can also be a creative enterprise, I would also suggest the following book: • "Creativity, Inc.: Overcoming Unseen Forces That Stand in the Way of True Inspiration" by Ed Catmull
- 26. Certs? When you need some credentials • Scrum? • Useful to have if you’re doing Scrum Master work. • Professional Scrum Master (PSM) by Scrum.org • No need to take a class. Higher minimal passing score needed. Scrum.org pushes more for understanding how to use scrum and adapt it to your requirements. • Certi fi ed Scrum Master (CSM) by Scrum Alliance • Requires you to take a class. Comparatively low passing score. The exam concentrates more on wrote knowledge of Scrum, and not as much how to apply or adapt it. • Agile • Anything potentially from PMI • If you don’t have the work history for a PMP • Try the CompTIA Project+ • Or do the Certi fi ed Associate in Project Management (CAPM) by PMI
- 27. Thank You Twitter: @admford Mastodon: @admford@infosec.exchange Linkedin: www.linkedin.com/in/danlagos/