Conf2013 bchristensen thebig_t

Infrastructure Operations


Monitoring for the big T.


About Me


3rd Generation Computer Geek
Roy Christensen: 35 years
3C > HoneyWell > Digital Equipment Corp
Mike Christensen: 40 years
DEC > Compaq> Hewlett Packard

Beau Christensen: 17 years
AstraZeneca > Send.com > DirecTV > Ping
Identity
92 years family experience (hah)

About Ping Identity
•

We believe secure professional and
personal identities underlie human
progress in a connected world. Our
purpose is to enable and protect
identity, defend privacy and secure the
Internet.

•

Over 1,000 companies, including over half
of the Fortune 100, rely on our awardwinning products to make the digital world
a better experience for hundreds of millions
of people.

•

Denver, Colorado. Est. 2003

Infrastructure Operations @Ping
Site Reliability Engineering (SRE)
Production Web Operations
Ops to On-Demand Dev
Configuration Engineering (CFE)
Automation, Deployment, Labs
Ops to On-Premise Dev
Infrastructure Engineering (IFE)
Iron, Network, Security
Ops to Support & Helpdesk


(Quickie) Architecture

•
•
•
•
•

Hybrid Cloud Application
VMware/AWS
SOA (lots of little services)
Red/Black Deployment
Cassandra & Galera MySQL


Splunk Systems

• 7 Indexers
• Distro search across 3 regional
data centers
• ~ 40Gib/day
• No clustering (yet)
• Universal FWD installed in all
templates
• Moving all Splunk to clustered
architecture in VPCs.


Trust
(the big T)


Because uptime is
dead.

http://www.kitchensoap.com/2013/01/03/availability-nuance-as-a-service/

http://uptime.pingidentity.com


What Builds Trust?

Lots of Things…

Security

Honesty

Reliability
Integrity

Dependability
Maturity

Transparency

We rolled those up into two.

Reliability & Transparency


Reliability:
Keep Shit Running

Keeping Shit Running
•

Architecture
–
–
–

•

Tools
–
–
–

•

SOA (duh!!!!)
Highly Automated Deployments
Active/Active, quick failover, multi-region
We use a ton of tools
We constantly question the tools we use
We are always looking for new tools

Security
–
–
–

Eyes always on
Scanners running continuously
Constant Remediation


Transparency:
Talk About Shit

Talking About Shit
•

Talk Publicly
–
–
–
–

•

When Stuff Breaks, Talk More
–
–
–

•

Talk about decisions we make
Talk about architecture
Talk to vendors
Talk to customers

Quick & dirty status updates
Verbose post mortems
Be honest

Metrics
–
–
–

Use 3rd Parties
Expose as much monitoring as you can
Make it relevant to customers


T=r+t
Trust = reliability +
transparency


Trust is a Zero Sum Game
Time
Trust


T = +2

Time
Trust


Time
Trust

T = -2

Time
Trust

T=0


Monitoring is the
foundation of
Trust.

is the backbone of
our monitoring
strategy.

<3 Monitoring
The right tool for the right job = Lots of tools.


Lots of tools = Best of Breed, or
Stack Monitoring


Let’s build the stack.


Transport & Network Layer.
Fundamental interactions between services.
Extremely important to understand, forms the
basis of troubleshooting efforts.

“Can you ping it? Tracert? Tcpdump?”


Instance Resource
Monitoring

Historical resource consumption, collected
and correlated with incidents and system
events. Scalability intelligence.
“Traditional” systems monitoring. Nagios,
Zenoss, Zabbix, etc.


Machine Data, Logs.
Extraordinary versatility: events, alerting,
reporting, security, business metrics, and
OI. Allows the creation of dashboards and
event types relative to your own
environment.
Events, logs, traditional “syslog stuff.”

Application Performance
Monitoring
Detailed view of how the applications are
running on top of the rest of the stack.
Identify bottlenecks, architecture issues,
and accurately model user experience with
RUM.
“Page loads are slow.” “Why are there SQL
queries that return 30,000 rows?”


Availability Monitoring
External agents monitor services from
multiple locations around the globe. Use
application heartbeats, not ICMP or TCP.
Provides near real-time alerting and is
immediately visible to the customer.
“The site is down.”


Splunk is the system’s mortar.
Binding systems together, filling gaps,
creating stability.


Security
Quick visual impact. Easy to
identify location, frequency,
and network address of
aggressor. Quick drill down
into detailed Splunk
searches.


Traffic
Real-time traffic maps are
immediately recognizable to
everyone in the
organization.


Operations
Operational dashboards provide
a quick overview of system
health by displaying error
correlation with traffic levels and
event type heat maps.


Global Load Balance
Shows traffic distribution between
production data centers.


Maximize Machine Data Layer Versatility
Build your own dashboards (it’s easy).
Maintain proper naming conventions:
na-den-www-13-8-17-0ec5e8da-128-193
Design for wetware.

Ask for feedback.
Maintain event types.


Having tons of tools is great. Each layer of the
stack is matched well with the system’s
requirements…

Aggregation of metrics is the new challenge.
How do you see it all?


Stack Integration


Stack Integration = MTTR
Speed
eventtype=critical.*

Application integration allows speedier access to event timeline and context of
the alarm using markers. Allows SRE to gather relevant information quicker,
using the same tools with less screens.

Splunk has a TON of
integrations.


See? -->


Always Need More.


Remember…

T=r+t

Keep Shit Running.


Talk About Shit.


Use Splunk.


Questions?


Thanks!
bchristensen@pingidentity.com
@beauchristensen
www.pingidentity.com/blogs
http://status.pingidentity.com


Conf2013 bchristensen thebig_t

More Related Content

What's hot (20)

Viewers also liked (16)

Similar to Conf2013 bchristensen thebig_t (20)

Recently uploaded (20)

Conf2013 bchristensen thebig_t