SlideShare a Scribd company logo
The Problem of Becoming a 3rd-Line Support Team
(& Why Swarming Works better for DevOps!)
Jon Hall
Principal Product Manager, BMC
@jonhall_
DevOps Enterprise Summit 2018 – Las Vegas
The people that excel and create the most value
are the ones that step outside their box
Hank Barnes, Gartner: “Playing Outside Your Box” May 2018.
LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
LEVEL 1 SUPPORT
Classic “Tiered” Support Structure
@jonhall_
Escalation
Escalation
Deconstructing the “Tiered” Support Structure
LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 1 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
@jonhall_
…when the answer is here… …or here.
Issues may spend time here
LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 1 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
@jonhall_
LEVEL 1 SUPPORT
LEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS
When tickets
eventually
escalate…
…they frequently
bounce back for
clarification
@jonhall_
LEVEL 1 SUPPORT
LEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS
LEVEL 1 SUPPORT
LEVEL 3 SPECIALISTS
LEVEL 2 SUPPORT
SUBJECT MATTER EXPERT
The system encourages “heroes” (not in a good way)
@jonhall_
Removing the tiers of support, and calling on the
collective expertise of a “swarm” of analysts.
Swarming defined
@jonhall_
Swarming
Network
Collaborative
Dynamic, loopy
Measured by value creation
Tiered support
Siloes and hierarchies
Directed
Linear, rigid
Measured on activity
24 hours, 365 days.
500 specialists, 3000 years of experience
200,000+ incidents addressed each year
10,000+ customers
Remedy, Control-M, TrueSight, etc…
Communication skills are a hiring focus
BMC Contact Centres
Support Centres
Support Centres Co-located with R&D
Pleasanton/
Sunnyvale
Houston
Austin
McLean/
Herndon
Lexington
Sao Paulo
Buenos Aires
Spain
Dublin
Winnersh
Amsterdam
Paris
Tel Hai
Pune
Singapore
Shanghai
Beijing
Seoul
Dalian
Tokyo
Melbourne
Houston, TX, USA
Dublin, Ireland
Dalian China
BMC Customer Support
@jonhall_
Swarming at BMC
Dispatch SwarmSeverity 1 Swarm Backlog Swarm
@jonhall_
Severity 1 Swarm
Prioritise
Swarming Process at BMC
@jonhall_
• Rapid responders
• Three agents, one week rotation
• Primary focus: Provide immediate response, resolve ASAP
Swarm lead
Communications
Other members
Research, coordinate, test
Severity 1 Swarm
@jonhall_
Severity 1 Swarm
Local Dispatch Swarm
Prioritise
30% solved here
Swarming Process at BMC
@jonhall_
• “Cherry pickers”
• Meet every 60-90 minutes
• Primary focus: Can new tickets be resolved immediately?
• Also: Validation of ticket details before assignment to specialists
Experienced analyst Less-experienced analyst
Dispatch Swarm
@jonhall_
Local Product-Line
Support Teams
Severity 1 Swarm
Local Dispatch Swarm
Prioritise
Swarming Process at BMC
@jonhall_
Local Product Line
Support Teams
Severity 1
Swarm
Local Dispatch Swarm
Prioritise
Severity 1
Swarm
Local Dispatch Swarm
Prioritise
Local Product Line
Support Teams
Swarming Process at BMC
@jonhall_
Local Product Line Support Teams Local Product Line Support Teams
Backlog Swarm Backlog Swarm Backlog Swarm
Swarming Process at BMC
@jonhall_
• Global fixers of troublesome tickets
• Meet regularly (often multiple times daily)
• Primary focus: Challenging tickets brought by local support teams
• Replaces inter-team and individual reassignments
Experienced analysts R&D Engineers
Backlog Swarms
@jonhall_
• Guidelines, not rules
• Metrics had to change (Swarming breaks traditional ones!)
• Supported people who became newly customer facing
• Banned ticket tennis and direct escalations to experts
• New tooling practices, particularly mobile and chat
Making it work at BMC Customer Support
@jonhall_
• 25% median resolution time improvement
• Customer satisfaction up 8 points
• More issues closed in <2 days
• Significant reduction in backlogs
• Halved on-boarding time
• Freed resources for innovative offerings
Results at BMC
@jonhall_
“Swarming works better than conventional processes.
I am able to get multiple experiences from swarm attendees
of similar cases they have worked, and what they did.
If there are no experiences, then it’s perspectives: Decades
of experience, providing guidance of how to troubleshoot”
- Senior Support Analyst, BMC
@jonhall_
“I have probably doubled my knowledge of the
products in a year because of Swarming,
and I have been here a long time”
- Senior Support Analyst, BMC
@jonhall_
Ford Connected Vehicles Division
Challenge: how to scale support from 275,000 cars, to
180+ million new vehicles every year.
• “You’ve got to go where people are” – Chad Jolly, Developer
• Tiered support would mean 4-5 days to get to the right team
• First Responders instigate and coordinate ad-hoc swarms for big issues
• Other teams have 1 person on rotation for swarming
• Swarm may get bigger over time as necessary and might include
engineers from Amazon, Microsoft, etc.
@jonhall_
• Costs may increase even as other metrics improve
• Difficult to evaluate individual contribution
• Organizing across time zones may be a challenge
• A few individuals sometimes dominate
• Finding the right people for a swarm is difficult
It’s not all positive!
Problems reported by some Swarming adopters
@jonhall_
“IT organizations that have tried to custom-adjust
current tools to meet DevOps practices have a
failure rate of 80%”
DevOps and the Cost of Downtime: Fortune 1000 Best Practice Metrics Quantified (IDC, 2014)
So… what does this all have to do with DevOps?
@jonhall_
• New services and applications suddenly appear
• More home-grown software
• Developers work in different tools
• New kinds of customer, especially external
DevOps challenges ServiceDesk orthodoxies…
@jonhall_
• Provision of support at industrial scale
• Adaptation to life "on call”
• Multi-cloud; Blend of old and new systems
• Customer/business context
• What to prioritize? Fix or build?
…but enterprise realities challenge DevOps
DevOps challenges ServiceDesk orthodoxies…
@jonhall_
“The enterprise space doesn’t move slowly
because they’re stupid, or they hate technology.
It’s because they have users”
—Luke Kanies, Puppet Founder, Configuration Management Camp 2015, Belgium.
@jonhall_
• Work-in-progress queues
• Asynchronous communication
• Single role teams
• Individual over-exposure
• Lack of knowledge sharing
How to annoy a DevOps practitioner
@jonhall_
LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
LEVEL 1 SUPPORT
Uh oh…
@jonhall_
Swarming aligns really well to DevOps
• Autonomy and self-organisation
• Knowledge transfer and skills development
• ChatOps, not email
• Prevention of accumulation of queued work
• Protection of individuals from burnout
@jonhall_
• Cynefin (Pronounced “kuh-nev-in”)
• Developed by Dave Snowden at IBM in 1999
• Taken independent in 2005
• “Signifies the multiple factors in our environment and
our experience that influence us in ways we can
never understand”
Exploring further: Swarming to deliver Cynefin
@jonhall_
@jonhall_
• Obvious and Complicated domains:
• Repeating relationship between cause and effect
• With Complicated you need to do analysis to find
that relationship
• Complex domain:
• Understanding the problem requires
experimentation and analysis.
• May, over time, be able to move to Complicated
• Chaotic domain:
• Dramatic and unconstrained
• Focus on damage limitation, try to move to
another domain
“Obvious” Domain
@jonhall_
• “Sense, Categorise, Respond”
• Template/knowledge-driven resolution
• Self service
“Complicated” Domain
@jonhall_
• “Sense, Analyse, Respond”
• Dispatch-type swarm – pair agents with varied experience
• Capture detailed knowledge for organizational learning
“Complex” Domain
• “Probe, Sense, Respond”
@jonhall_
“Complex” Domain
• “Probe, Sense, Respond”
@jonhall_
“Chaotic” Domain
• “Act, Sense, Respond”
• Sub-swarms
• Deal with the acute situation
• Try to discover sufficient
information to move to complex
@jonhall_
• Service Management needs to evolve its practices and
tooling to better position its value to DevOps teams. We
need your help to do this right.
• We’d like to listen to how support is affecting your role,
as your impact grows in your enterprise.
• You are agents of change in enterprises, with a good
opportunity to influence thinking.
What next?
@jonhall_
medium.com/@jonhall_serviceinnovation.org/intelligent-swarming
Some more information

More Related Content

PPTX
Service Manager Dag, Netherlands 2018: Why we should ditch the 3-tier support...
PPTX
DevOps Enterprise Summit 2019 - How Swarming Enables Enterprise Support to wo...
PPTX
Velocity19 Berlin: Swarming, Cynefin… and avoiding the problems of becoming a...
PPTX
Devopsdays Edinburgh 2017 - Ignite talk - Swarming
PPTX
SRVision 2019, Utrecht: Swarming and Cynefin
PPTX
Devops In The Enterprise: How Swarming Can Fix The Problem Of Becoming A 3rd-...
PPTX
Support at scale in a DevOps world How Swarming and Cynefin can save you from...
PPTX
ITSM, Swarming and Devops
Service Manager Dag, Netherlands 2018: Why we should ditch the 3-tier support...
DevOps Enterprise Summit 2019 - How Swarming Enables Enterprise Support to wo...
Velocity19 Berlin: Swarming, Cynefin… and avoiding the problems of becoming a...
Devopsdays Edinburgh 2017 - Ignite talk - Swarming
SRVision 2019, Utrecht: Swarming and Cynefin
Devops In The Enterprise: How Swarming Can Fix The Problem Of Becoming A 3rd-...
Support at scale in a DevOps world How Swarming and Cynefin can save you from...
ITSM, Swarming and Devops

What's hot (20)

PPTX
SDI19: Swarming and Devops for ITSM
PPTX
Configuration Management Camp 2018: The problem of becoming "3rd line support...
PPTX
devopsdays Stockholm Ignite talk: Aligning DevOps with Enterprise-scale custo...
PPTX
Site Reliability Engineering: Harnessing (and redefining) it for ITSM
PPTX
DevOpsDays Riga - Swarming Presentation
PDF
Jan de Vries - Becoming antifragile is more important than ever in disruptive...
PDF
Deep learning - a primer
PDF
The 7 quests of resilient software design
PDF
Digitization solutions - A new breed of software
PDF
DevOps Paradox: Going Faster Brings Higher Quality, Lower Costs, & Better Out...
PDF
DevOps: The Future is Already Here — It’s Just Unevenly Distributed
PDF
Production-ready Software
PDF
Brighttalk high scale low touch and other bedtime stories - final
PDF
Site Reliability Engineering (SRE) - Tech Talk by Keet Sugathadasa
PDF
Adapting Scrum in an Organization with Tailored Processes
PPT
2012 Velocity London: DevOps Patterns Distilled
PDF
Mary Poppendieck: The Aware Organization - Lean IT Summit 2014
PPTX
SecureWorld: Security is Dead, Rugged DevOps 1f
PPTX
2.) services (people &amp; culture)
PPTX
QCon 2014 - Principles of Reliable Communication
SDI19: Swarming and Devops for ITSM
Configuration Management Camp 2018: The problem of becoming "3rd line support...
devopsdays Stockholm Ignite talk: Aligning DevOps with Enterprise-scale custo...
Site Reliability Engineering: Harnessing (and redefining) it for ITSM
DevOpsDays Riga - Swarming Presentation
Jan de Vries - Becoming antifragile is more important than ever in disruptive...
Deep learning - a primer
The 7 quests of resilient software design
Digitization solutions - A new breed of software
DevOps Paradox: Going Faster Brings Higher Quality, Lower Costs, & Better Out...
DevOps: The Future is Already Here — It’s Just Unevenly Distributed
Production-ready Software
Brighttalk high scale low touch and other bedtime stories - final
Site Reliability Engineering (SRE) - Tech Talk by Keet Sugathadasa
Adapting Scrum in an Organization with Tailored Processes
2012 Velocity London: DevOps Patterns Distilled
Mary Poppendieck: The Aware Organization - Lean IT Summit 2014
SecureWorld: Security is Dead, Rugged DevOps 1f
2.) services (people &amp; culture)
QCon 2014 - Principles of Reliable Communication
Ad

Similar to DevOps Enterprise Summit Las Vegas 2018: The Problem of Becoming a 3rd-Line Support Team (& Why Swarming Works better for DevOps!) (20)

PPTX
Swarming: How a new approach to support can save DevOps teams from 3rd-line t...
PPTX
DevOpsDaysRiga 2018: Jon Hall - DevOps in the enterprise: how "swarming" can ...
PPTX
SITS15: Swarming - A radical new way to deliver service
PDF
Self-Service Operations: Because Ops Still Happens
PPTX
Lucas Gravley - HP - Self-Healing And Monitoring in a DevOps world
PDF
Operations as a Service: Because Failure Still Happens
PDF
Strategies for building, managing, and scaling technology teams
PPTX
Is DevOps Really Changing IT Support?
PDF
Dev ops lessons learned - Michael Collins
PDF
SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps
PDF
How to Avoid Cloud Confusion, DevOps dilemma, Microservice Madness
PPTX
Bring Down The Walls for Confusion - Agile and Beyond 2016
PDF
The History of DevOps (and what you need to do about it)
PDF
Self-Service Operations: Because Failure Still Happens (Developer Edition)
PPTX
THE RISE AND FALL OF SERVERLESS COSTS - TAMING THE (SERVERLESS) BEAST
PDF
DevOps 101 - DevOps Columbia 3-20-2025.pdf
PPTX
DevOps for Dinosaurs
PPSX
PPTX
DevOps topologies
PPTX
Intro to DevOps
Swarming: How a new approach to support can save DevOps teams from 3rd-line t...
DevOpsDaysRiga 2018: Jon Hall - DevOps in the enterprise: how "swarming" can ...
SITS15: Swarming - A radical new way to deliver service
Self-Service Operations: Because Ops Still Happens
Lucas Gravley - HP - Self-Healing And Monitoring in a DevOps world
Operations as a Service: Because Failure Still Happens
Strategies for building, managing, and scaling technology teams
Is DevOps Really Changing IT Support?
Dev ops lessons learned - Michael Collins
SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps
How to Avoid Cloud Confusion, DevOps dilemma, Microservice Madness
Bring Down The Walls for Confusion - Agile and Beyond 2016
The History of DevOps (and what you need to do about it)
Self-Service Operations: Because Failure Still Happens (Developer Edition)
THE RISE AND FALL OF SERVERLESS COSTS - TAMING THE (SERVERLESS) BEAST
DevOps 101 - DevOps Columbia 3-20-2025.pdf
DevOps for Dinosaurs
DevOps topologies
Intro to DevOps
Ad

More from Jon Stevens-Hall (14)

PPTX
Expanding our Understanding: Complex Adaptive Systems
PPTX
Rethinking Site Reliability Engineering for ITSM - SDI virtual event "New Way...
PPTX
BMC Engage 2015: Optimizing Service Desk Interactions with Knowledge Management
PPTX
Knowledge Management in BMC Remedy 9.1
PPTX
How the Internet of Things and 20 billion devices will change your job
PPTX
IAITAM ACE 2016, New Orleans - Presentation
PPTX
Evolving Service for the Digital Workplace
PPTX
Optimizing Service Desk Interactions with Knowledge Management - BMC Engage 2015
PPTX
BMC Engage 2015: IT Asset Management - An essential pillar for the digital en...
PPTX
BMC Engage 2015: Smart IT, MyIT and the Power of the Service Platform
PPTX
IT Trends Set to Shape Software Asset Management (IBSMA SAM Summit June 2015)
PPTX
Bridging the Gap - The Value of Integrated Asset and Service Management
PPTX
BMC Engage - ITAM 2015-2020: The Evolving Role of the IT Asset Manager
PPTX
Bridging the Gap - the Value of Integrated Asset and Service Management
Expanding our Understanding: Complex Adaptive Systems
Rethinking Site Reliability Engineering for ITSM - SDI virtual event "New Way...
BMC Engage 2015: Optimizing Service Desk Interactions with Knowledge Management
Knowledge Management in BMC Remedy 9.1
How the Internet of Things and 20 billion devices will change your job
IAITAM ACE 2016, New Orleans - Presentation
Evolving Service for the Digital Workplace
Optimizing Service Desk Interactions with Knowledge Management - BMC Engage 2015
BMC Engage 2015: IT Asset Management - An essential pillar for the digital en...
BMC Engage 2015: Smart IT, MyIT and the Power of the Service Platform
IT Trends Set to Shape Software Asset Management (IBSMA SAM Summit June 2015)
Bridging the Gap - The Value of Integrated Asset and Service Management
BMC Engage - ITAM 2015-2020: The Evolving Role of the IT Asset Manager
Bridging the Gap - the Value of Integrated Asset and Service Management

Recently uploaded (20)

PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PDF
Empathic Computing: Creating Shared Understanding
PDF
Advanced IT Governance
PDF
KodekX | Application Modernization Development
PDF
Advanced Soft Computing BINUS July 2025.pdf
PDF
Unlocking AI with Model Context Protocol (MCP)
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PDF
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
PDF
Machine learning based COVID-19 study performance prediction
PDF
Review of recent advances in non-invasive hemoglobin estimation
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PPTX
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
PDF
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Empathic Computing: Creating Shared Understanding
Advanced IT Governance
KodekX | Application Modernization Development
Advanced Soft Computing BINUS July 2025.pdf
Unlocking AI with Model Context Protocol (MCP)
CIFDAQ's Market Insight: SEC Turns Pro Crypto
Reach Out and Touch Someone: Haptics and Empathic Computing
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
Machine learning based COVID-19 study performance prediction
Review of recent advances in non-invasive hemoglobin estimation
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Diabetes mellitus diagnosis method based random forest with bat algorithm
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
“AI and Expert System Decision Support & Business Intelligence Systems”
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
Chapter 3 Spatial Domain Image Processing.pdf
Mobile App Security Testing_ A Comprehensive Guide.pdf
Advanced methodologies resolving dimensionality complications for autism neur...
Per capita expenditure prediction using model stacking based on satellite ima...

DevOps Enterprise Summit Las Vegas 2018: The Problem of Becoming a 3rd-Line Support Team (& Why Swarming Works better for DevOps!)

  • 1. The Problem of Becoming a 3rd-Line Support Team (& Why Swarming Works better for DevOps!) Jon Hall Principal Product Manager, BMC @jonhall_ DevOps Enterprise Summit 2018 – Las Vegas
  • 2. The people that excel and create the most value are the ones that step outside their box Hank Barnes, Gartner: “Playing Outside Your Box” May 2018.
  • 3. LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 1 SUPPORT Classic “Tiered” Support Structure @jonhall_
  • 4. Escalation Escalation Deconstructing the “Tiered” Support Structure LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT LEVEL 1 SUPPORT LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS @jonhall_
  • 5. …when the answer is here… …or here. Issues may spend time here LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT LEVEL 1 SUPPORT LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS @jonhall_
  • 6. LEVEL 1 SUPPORT LEVEL 2 SUPPORT LEVEL 3 SPECIALISTS When tickets eventually escalate… …they frequently bounce back for clarification @jonhall_
  • 7. LEVEL 1 SUPPORT LEVEL 2 SUPPORT LEVEL 3 SPECIALISTS LEVEL 1 SUPPORT LEVEL 3 SPECIALISTS LEVEL 2 SUPPORT SUBJECT MATTER EXPERT The system encourages “heroes” (not in a good way) @jonhall_
  • 8. Removing the tiers of support, and calling on the collective expertise of a “swarm” of analysts. Swarming defined @jonhall_ Swarming Network Collaborative Dynamic, loopy Measured by value creation Tiered support Siloes and hierarchies Directed Linear, rigid Measured on activity
  • 9. 24 hours, 365 days. 500 specialists, 3000 years of experience 200,000+ incidents addressed each year 10,000+ customers Remedy, Control-M, TrueSight, etc… Communication skills are a hiring focus BMC Contact Centres Support Centres Support Centres Co-located with R&D Pleasanton/ Sunnyvale Houston Austin McLean/ Herndon Lexington Sao Paulo Buenos Aires Spain Dublin Winnersh Amsterdam Paris Tel Hai Pune Singapore Shanghai Beijing Seoul Dalian Tokyo Melbourne Houston, TX, USA Dublin, Ireland Dalian China BMC Customer Support @jonhall_
  • 10. Swarming at BMC Dispatch SwarmSeverity 1 Swarm Backlog Swarm @jonhall_
  • 11. Severity 1 Swarm Prioritise Swarming Process at BMC @jonhall_
  • 12. • Rapid responders • Three agents, one week rotation • Primary focus: Provide immediate response, resolve ASAP Swarm lead Communications Other members Research, coordinate, test Severity 1 Swarm @jonhall_
  • 13. Severity 1 Swarm Local Dispatch Swarm Prioritise 30% solved here Swarming Process at BMC @jonhall_
  • 14. • “Cherry pickers” • Meet every 60-90 minutes • Primary focus: Can new tickets be resolved immediately? • Also: Validation of ticket details before assignment to specialists Experienced analyst Less-experienced analyst Dispatch Swarm @jonhall_
  • 15. Local Product-Line Support Teams Severity 1 Swarm Local Dispatch Swarm Prioritise Swarming Process at BMC @jonhall_
  • 16. Local Product Line Support Teams Severity 1 Swarm Local Dispatch Swarm Prioritise Severity 1 Swarm Local Dispatch Swarm Prioritise Local Product Line Support Teams Swarming Process at BMC @jonhall_
  • 17. Local Product Line Support Teams Local Product Line Support Teams Backlog Swarm Backlog Swarm Backlog Swarm Swarming Process at BMC @jonhall_
  • 18. • Global fixers of troublesome tickets • Meet regularly (often multiple times daily) • Primary focus: Challenging tickets brought by local support teams • Replaces inter-team and individual reassignments Experienced analysts R&D Engineers Backlog Swarms @jonhall_
  • 19. • Guidelines, not rules • Metrics had to change (Swarming breaks traditional ones!) • Supported people who became newly customer facing • Banned ticket tennis and direct escalations to experts • New tooling practices, particularly mobile and chat Making it work at BMC Customer Support @jonhall_
  • 20. • 25% median resolution time improvement • Customer satisfaction up 8 points • More issues closed in <2 days • Significant reduction in backlogs • Halved on-boarding time • Freed resources for innovative offerings Results at BMC @jonhall_
  • 21. “Swarming works better than conventional processes. I am able to get multiple experiences from swarm attendees of similar cases they have worked, and what they did. If there are no experiences, then it’s perspectives: Decades of experience, providing guidance of how to troubleshoot” - Senior Support Analyst, BMC @jonhall_
  • 22. “I have probably doubled my knowledge of the products in a year because of Swarming, and I have been here a long time” - Senior Support Analyst, BMC @jonhall_
  • 23. Ford Connected Vehicles Division Challenge: how to scale support from 275,000 cars, to 180+ million new vehicles every year. • “You’ve got to go where people are” – Chad Jolly, Developer • Tiered support would mean 4-5 days to get to the right team • First Responders instigate and coordinate ad-hoc swarms for big issues • Other teams have 1 person on rotation for swarming • Swarm may get bigger over time as necessary and might include engineers from Amazon, Microsoft, etc. @jonhall_
  • 24. • Costs may increase even as other metrics improve • Difficult to evaluate individual contribution • Organizing across time zones may be a challenge • A few individuals sometimes dominate • Finding the right people for a swarm is difficult It’s not all positive! Problems reported by some Swarming adopters @jonhall_
  • 25. “IT organizations that have tried to custom-adjust current tools to meet DevOps practices have a failure rate of 80%” DevOps and the Cost of Downtime: Fortune 1000 Best Practice Metrics Quantified (IDC, 2014) So… what does this all have to do with DevOps? @jonhall_
  • 26. • New services and applications suddenly appear • More home-grown software • Developers work in different tools • New kinds of customer, especially external DevOps challenges ServiceDesk orthodoxies… @jonhall_
  • 27. • Provision of support at industrial scale • Adaptation to life "on call” • Multi-cloud; Blend of old and new systems • Customer/business context • What to prioritize? Fix or build? …but enterprise realities challenge DevOps DevOps challenges ServiceDesk orthodoxies… @jonhall_
  • 28. “The enterprise space doesn’t move slowly because they’re stupid, or they hate technology. It’s because they have users” —Luke Kanies, Puppet Founder, Configuration Management Camp 2015, Belgium. @jonhall_
  • 29. • Work-in-progress queues • Asynchronous communication • Single role teams • Individual over-exposure • Lack of knowledge sharing How to annoy a DevOps practitioner @jonhall_
  • 30. LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 1 SUPPORT Uh oh… @jonhall_
  • 31. Swarming aligns really well to DevOps • Autonomy and self-organisation • Knowledge transfer and skills development • ChatOps, not email • Prevention of accumulation of queued work • Protection of individuals from burnout @jonhall_
  • 32. • Cynefin (Pronounced “kuh-nev-in”) • Developed by Dave Snowden at IBM in 1999 • Taken independent in 2005 • “Signifies the multiple factors in our environment and our experience that influence us in ways we can never understand” Exploring further: Swarming to deliver Cynefin @jonhall_
  • 33. @jonhall_ • Obvious and Complicated domains: • Repeating relationship between cause and effect • With Complicated you need to do analysis to find that relationship • Complex domain: • Understanding the problem requires experimentation and analysis. • May, over time, be able to move to Complicated • Chaotic domain: • Dramatic and unconstrained • Focus on damage limitation, try to move to another domain
  • 34. “Obvious” Domain @jonhall_ • “Sense, Categorise, Respond” • Template/knowledge-driven resolution • Self service
  • 35. “Complicated” Domain @jonhall_ • “Sense, Analyse, Respond” • Dispatch-type swarm – pair agents with varied experience • Capture detailed knowledge for organizational learning
  • 36. “Complex” Domain • “Probe, Sense, Respond” @jonhall_
  • 37. “Complex” Domain • “Probe, Sense, Respond” @jonhall_
  • 38. “Chaotic” Domain • “Act, Sense, Respond” • Sub-swarms • Deal with the acute situation • Try to discover sufficient information to move to complex @jonhall_
  • 39. • Service Management needs to evolve its practices and tooling to better position its value to DevOps teams. We need your help to do this right. • We’d like to listen to how support is affecting your role, as your impact grows in your enterprise. • You are agents of change in enterprises, with a good opportunity to influence thinking. What next? @jonhall_

Editor's Notes

  • #28: In a company of 4000 people, things can get out of hand really fast if you don't have customer context” “If you're dropped in the middle of something, how did you get here?”\ “The person who is on call at 4am needs to know who has been doing what”