thwackCamp 2013: Cut the Alert Noise: Best Practices to Avoid Common Pitfalls and Optimize Managing IT Alers

Cut the Alert Noise: Best Practices to Avoid
Common Pitfalls and Optimize Managing IT Alerts
SolarWinds® thwackCamp 2013
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

Agenda
» What is Alert Central™ and how does it work?
» Solving common IT alert problems with Alert Central
 Automatic assignment and escalation
 Reassignment, moving alerts between team members/groups quickly
 Integrating on call scheduling with alert management
 Filtering out alerts that don’t need escalation
 Preventing alerts from duplicate systems from being escalated

How Alert Central Works
» Deployed as Virtual Appliance
» Alerts from IT systems company-wide are
consolidated in Alert Central (collected via e-mail
or direct integration with SolarWinds Orion family
products)
» Alert Central handles escalation and on-call
scheduling (with configurable policies) to ensure
alerts go to the appropriate person
» That person can then acknowledge and clear the
alert, reassign, or escalate it (via email or in
console)

Automatic Assignment & Escalation
» Problem: get alerts ONLY to the people who can handle them.
» Problem: if someone’s not available, alerts can get stuck until they are.
» Problem: if everyone’s notified, it takes time to identify whether they care.
» Solution:
 Create groups for each logical staff group in Alert Central.
 Create escalation policies within each group to follow the group’s notification preferences.
 Assign alerts to each group based on information in the alert.

Moving alerts between team members/groups quickly
» Problem: With a mass email system, it’s easy to lose track of who is responsible or
miss that it’s you that got “assigned” the alert.
» Problem: When alerts need to go to another group, it can be unclear how to do so.
» Problem: Working remotely or from home, having to log in to deal with alerts can be
time consuming.
» Solution:
 Automatic assignment makes sure only one person has the ball.
 Reassign alerts via email or the console
 Reassign alerts to either a specific user or a group, which will use the group’s configured
escalation policy automatically.
 Each user can configure notification policies in Alert Central for their notification preferences.

Integrating on call scheduling with alert management
» Problem: Maintaining an external on call calendar (on the whiteboard, in a
spreadsheet, in a shared calendar) can be hard to keep accurate
» Problem: Without on call integrated to alerting, either a shared device has to be
used or a person has to be available 24/7 to find the on call person
» Solution:
 Use on call calendars within Alert Central to support rotations, regular schedules
 Integrate on call directly with escalation policies – with fallback options

Filtering out alerts that don’t need escalation
» Problem: Some alerts don’t need immediate action or are handled by regular
helpdesk/IT staff, but come in through the same email source
» Problem: Realistically, systems sometimes generate noise or invalid alerts, and it’s
faster to tune them out until a fix is made
» Solution:
 Identify criteria for noise alerts based on the email/alert details
 In Alert Central’s source configuration, choose the “Trash this alert” option for that alert criteria

Preventing alerts from duplicate systems from being escalated
» Problem: Implementing new systems while leaving old systems running generates
some duplicate alerts
» Problem: Overlapping monitoring systems can generate similar alerts, but both
copies don’t need to be delivered
» Problem: In any case, it’s not possible to tune out an entire source, filtering needs to
be more sensitive
» Solution:
 Identify data in the duplicate alert that doesn’t need to be delivered
 Use “Trash this alert” to indicate the duplicate alerts should be skipped
 As more cases come up, continue evolving the policy easily

Resources
» Alert Central Getting Started Videos
 Configuring Escalation Policies & On Call:
http://www.youtube.com/watch?feature=player_embedded&v=8h7nUEXJ3ws
 Configuring Orion Alert Sources:
http://www.youtube.com/watch?feature=player_embedded&v=BJLK0IqDHII
 Configuring Email Alert sources:
http://www.youtube.com/watch?feature=player_embedded&v=oMxN2oZZM4s
» Alert Central on thwack®
 http://thwack.solarwinds.com/community/tools_tht/alert-central
» Alert Central Resource Library
 Links to FAQs, downloads, and more: http://thwack.solarwinds.com/docs/DOC-170671

Questions?

Thank You!
The SOLARWINDS and SOLARWINDS & Design marks are the exclusive property of SolarWinds
Worldwide, LLC, are registered with the U.S. Patent and Trademark Office, and may be registered or
pending registration in other countries. All other SolarWinds trademarks, service marks, and logos
may be common law marks, registered or pending registration in the United States or in other
countries. All other trademarks mentioned herein are used for identification purposes only and
may be or are trademarks or registered trademarks of their respective companies.

thwackCamp 2013: Cut the Alert Noise: Best Practices to Avoid Common Pitfalls and Optimize Managing IT Alers

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to thwackCamp 2013: Cut the Alert Noise: Best Practices to Avoid Common Pitfalls and Optimize Managing IT Alers (20)

More from SolarWinds (20)

Recently uploaded (20)

thwackCamp 2013: Cut the Alert Noise: Best Practices to Avoid Common Pitfalls and Optimize Managing IT Alers

Editor's Notes