Loggly - How to Scale Your Architecture and DevOps Practices for Big Data Applications (AWS DevOps Week 2017)

1© 2017 Loggly, Inc. Confidential & Proprietary. 6/29/17
How to Scale Your
Architecture and DevOps
Practices for Big Data
Applications
Manoj Chaudhary
CTO and VP of Engineering
March 30, 2017

• Cloud-based log management
• Founded in 2009
• Based in San Francisco
• 10,000+ customers
• Startups to Fortune 500

How log management starts
Management = simple stuff
• Rotate files, compress and delete
• Scan for specific events (not easy)
• Log retention policies evolve over
time

Log volume
Self-InflictedPain
“…hmmm, our logs are getting a bit bloated”
“…let’s spend time managing our log capacity”
“…how can I make this someone else’s problem!”
As log data grows

Focus on managing application vs. managing logs
from the trenches…
If you get a disk space alert, first login…
% sudo rm –rf /var/log/apache2/*
Admit it, we’ve all seen this kind of thing!
Do not make this is an either/or proposition!
!

We believe the future is all
about increasingly complex
systems, and companies need
better ways to be able to
understand them.

The way we make complex
systems understandable is by
making it ridiculously easy to
reveal the hidden stories in your
log data.

Loggly makes log
management easy and
simple
• Cloud-based SaaS for easy central log
collection, aggregation, management
• Agent-free, easy setup
• Uses open or industry standards and native
log transfer capabilities (example: syslog)
• No dependency on / maintenance effort
from proprietary agents
• 54 logging technologies supported out of
the box, list growing
• Setup Wizard or optionally fully manual
configuration with full control and
transparency

Dynamic parsing
• Parses data in real-time at ingestion
time, auto-recognizes common log
formats, but also supports custom
parsing rules
• Automatically breaks data down into
meaningful groups, fields, values that
can be mouse-navigated in a menu-
style structure (Dynamic Field
Explorer™)
• JSON support / extraction
• Custom parsing and tagging rules
allow to segregate dynamically
• Self-documenting data inventory

Gamut Search™
• Provides instant results over massive
amounts of data and long time
periods
• Rather than waiting minutes, hours or
days for a query to complete, the new
capabilities enable users to begin
analyzing the entire range, or the
“gamut” of their log data, immediately,
in a highly interactive fashion.
• No special query language to learn,
common Regular Expressions syntax
• Surround Search shows events context
with one single click

Other key features
• In-depth analytics
• Dashboards, pre-configured and
customizable, shareable
• Anomaly Detection
• Alerts that can be sent to HipChat,
Slack, PagerDuty, HTTP endpoints,
others
• JIRA Software integration, point-and-
click ticket creation without leaving
Loggly
Log management is
part of a bigger
process!
DevOps, Agile, CI/CD, …

Log management is a big data problem
Massive incoming event stream
Fundamentally multi-tenant
Scalable framework for analysis
Near real-time indexing
Near real-time search
Time-series index management
Logs are TLDR;
Traffic is unpredictable

But can be viewed as simple as
Ingest Process Index
Search and
other
Services

Let’s look under the covers of simplicity
Amazon
Route 53
Kafka
broker
Kafka
broker
Search and
analytics
engine
Ingestion pod Indexing pod
S3 ingestion
Services on processing pod
Services on indexing
Index
management
• Convert unstructured data to
structured
• Data processing happens here
• Checkpoint processed data to Kafka
• Can have multiple indexing clusters
• Time-series index management
• Index and store data in search and
analytics engine
• Entry point for logs
• Collect and validate
• Checkpoint logs to Kafka
Amazon S3 &
AWS CloudTrail
Amazon
RDS
MySQL
S3 archiving
Amazon S3

Scalability means not
just the ability to
operate, but to operate
efficiently and with
adequate quality of
service, over the given
range of configurations.
Build for scalability
developing
service
Scalability is a feature
“
”
+ running
service
governing
service+
SaaS engineering =

All customers don’t behave as good citizens at all times
“Noisy neighbors” with spikes in log volumes
• Application on fire
• Log management configuration problem
• Other human error
• Spikes can last a long time

How to mitigate the effects of unpredictable load patterns
• Have governors
• Hawk for your infrastructure
• Processes for managing
out-of-policy activity
• Set up metrics and alerts that let
you know about unexpected
behavior

Sits on top of platform and “watches” what’s going on – across tenants
Governors to segregate tenants
Identify out-of-
policy behavior
Segregate
misbehavior
Inform the
right people

Search and
analytics engine
Index management
Trusted platform for all customers
Amazon
Route53
Kafka broker
Kafka
broker
Services on indexing pod
Search and
analytics engine
Index management
Kafka broker
Ingestion pod Indexing pod
Services on ingestion pod like
S3 archiving
Services on processing pod
Governors
Amazon ElastiCache Amazon RDS MySQL
RDS
RDS

Case study

Troubleshooting use case
Problem
• Running our business involves integrating with
more than 100 web-based services and billions
of API calls every month.
• There are many places where things can go
awry. Segment depends on its log data to
troubleshoot operational issues
• Solve operational issues with partner
integrations
• Isolate relevant log data from billions of API calls
per month
Competing solutions
• ELK stack
• Homegrown log management system

Troubleshooting use case
Solution
• Use UUID to trace API calls to and from partner services
• Monitor API volume from specific customers
• Error monitoring with code releases
Results
• Faster MTTR
• DevOps team no longer involved in most troubleshooting = Devops efficiency
• Pro-active alerts = Devops efficiency and increase customer satisfaction.
• Integrated with bug tracking system = Devops and engineering team efficiency

Troubleshooting demo

Monitoring for errors

Other common Loggly use cases
Proactive log
monitoring and alerting
Data analysis and
optimization
Team collaboration

Manoj Chaudhary
twitter: @twitt_2_manoj
Visit us at loggly.com or follow @loggly on Twitter.
Try Loggly for Free! → https://www.loggly.com/

Loggly - How to Scale Your Architecture and DevOps Practices for Big Data Applications (AWS DevOps Week 2017)

More Related Content

What's hot (20)

Similar to Loggly - How to Scale Your Architecture and DevOps Practices for Big Data Applications (AWS DevOps Week 2017) (20)

Recently uploaded (20)

Loggly - How to Scale Your Architecture and DevOps Practices for Big Data Applications (AWS DevOps Week 2017)