Setting Up Sumo Logic - Apr 2017
Download as PPTX, PDF1 like457 views
This document outlines the setup and optimization strategies for Sumo Logic data collection, providing guidance on designing deployment strategies and implementing best practices for metadata naming conventions. It covers various collector options, data source types, and optimization tools available to enhance search performance. The document emphasizes the importance of a robust _sourcecategory naming convention and field extraction rules to effectively manage and analyze log data.
![Sumo Logic Confidential
Scheduled Views
Copies of subsets of data, similar to a relation DB materialized view.
Use Cases
Pre-aggregated data (e.g. for long-term trends)
Find the needle in the haystack….
Best Practices
We recommend selectivity of > 1:10000
How They Work
View is updated by service ~once a minute
Allows for backfilling
Search view using _view=[viewname]
Data does count against ingest volume](https://image.slidesharecdn.com/settingsumologicapr2017-170426234727/85/Setting-Up-Sumo-Logic-Apr-2017-22-320.jpg)
Setting Up Sumo Logic - Apr 2017
- 1. Sumo Logic Confidential Setting Up Sumo Logic Data Collection and System Optimization Mario Sanchez Director of Technical Content and Training April 2017 Welcome. To give everyone a chance to successfully connect, we’ll start at 10:05 AM Pacific. Note you are currently muted.
- 2. Sumo Logic Confidential At the completion of this webinar, you will be able to… Deploy a data collection strategy that best fits your environment Implement best practices around data collection Develop a robust naming convention for your metadata Learn to utilize optimization tools to enhance search performance
- 3. Sumo Logic Confidential What is Sumo Logic?
- 4. Sumo Logic Confidential Continuous Intelligence DEVOPS IT INFRASTRUCTURE AND OPERATIONS COMPLIANCE AND SECURITY DEVOPS Streamline continuous delivery Monitor KPI’s and Metrics Accelerate Troubleshooting IT INFRASTRUCTURE AND OPERATIONS Monitor all workloads Troubleshoot and increase uptime Simplify, Modernize, and save costs COMPLIANCE AND SECURITY Automate and demonstrate compliance Audit all systems Think beyond rules Sumo Logic Cloud Analytics Service
- 5. Sumo Logic Confidential Enterprise Logs are Everywhere Custom App Code Server / OS Virtual Databases Network Open Source Middleware Content Delivery IaaS, PaaS SaaS Security
- 6. Sumo Logic Confidential High-Level Data Flow
- 7. Sumo Logic Confidential Sumo Logic Data Flow Data Collection Search & Analyze Visualize & Monitor Alerts Dashboards Collectors Sources Operators Detect 1 2 3
- 8. Sumo Logic Confidential Data Collection Strategy
- 9. Sumo Logic Confidential Designing Your Deployment • Sumo Logic Data Collection is infinitely flexible. • Design a Sumo Logic deployment that's right for your organization. • Installed versus Hosted Collectors.
- 10. Sumo Logic ConfidentialSumo Logic Confidential Collector and Deployment Options Collector Cloud Data Collection Centralized Data Collection Local Data Collection Collector CollectorCollector Collector Hosted Collectors Installed Collectors Best Practices on Designing Your Deployment
- 11. Sumo Logic Confidential Local Data Collection The Sumo Logic Collector is installed on all target Hosts and, where possible, sends log data produced on those target Hosts directly to Sumo Logic Backend via https connection. Source Types Local Files Operating Systems, Middleware, Custom Apps, etc. Windows Events Local Windows Events Docker Logs and Stats Syslog (dedicated Collector) Network Devices, Snare, etc Script (dedicated Collector) Cloud API’s, Database Content, binary data Typical Scenarios Customers with large amounts of (similar) servers, using orchestration/automation, mostly OS and application logs - On Premise Datacenters - Cloud Instances Benefits/Drawbacks + No Hardware Requirement + Automation (Chef/Puppet/Scripting) - Outbound Internet Access Required - Resource Usage on Target
- 12. Sumo Logic Confidential Source Types Syslog Operating Systems, Middleware, Custom Applications, etc Windows Events Remote Windows Events Script Cloud API’s, Database Content, binary data Typical Scenarios Customers with mostly Windows Environments or existing logging infrastructure (syslog/logstash) - On Premise Datacenters Benefits/Drawbacks + No Outbound Internet Access + Leverage existing logging Infrastructure - Scale - Dedicated Hardware - Complexity (Failover, syslog rules) Centralized Data Collection The Sumo Logic Collector is installed on a set of dedicated machines, these collect log data from the target Hosts via various remote mechanisms and forward the data to the Sumo Logic Backend. This can be accomplished by either using Sumo Logic syslog source type or by running Syslog Servers (syslog-ng, rsyslog), write to file, and collect from there.
- 13. Sumo Logic Confidential Source Types S3 Bucket Any data written to S3 buckets (AWS Audit or other) HTTPS Lambda Scripts, Akamai, One Login, Log Appender Libraries, etc. Google / O365 Google API and O365 API Typical Scenarios Customers using Cloud Infrastructure, while it's possible to rely on Cloud Data Collection entirely, this is not typical. These source types are normally just part of the overall collection strategies Benefits/Drawbacks + No Software Installation - S3 Latency issues - Https Post Caching Need Cloud Data Collection Most Data is generated in the Cloud and by Cloud Services and is collected via Sumo Logics Cloud Integrations.
- 14. Sumo Logic Confidential Metadata Design
- 15. Sumo Logic Confidential What is Metadata? Tag Description _collector Name of the collector (defaults to hostname) _source Name of the source this data came through _sourceHost Hostname of the server (defaults to hostname) _sourceName Name and Path of the log file _sourceCategory Can be freely configured. Main metadata tag Metadata tags are associated with each log message that is collected. Values are set through collector and source configuration.
- 16. Sumo Logic ConfidentialSumo Logic Confidential Source Category Best Practices Recommended nomenclature for Source Categories Component1/Component2/Component3… From least descriptive to most descriptive * Note: Not all types of logs need to have the same amount of levels. Best Practices: Good Source Category, Bad Source Category Prod/MyApp1/Apache/Access Prod/MyApp1/Apache/Error Prod/MyApp1/CloudTrail Dev/MyApp1/Apache/Access Dev/MyApp1/Apache/Error Dev/MyApp1/CloudTrail Prod/MyApp2/Nginx/Access Prod/MyApp2/Tomcat/Access Prod/MyApp2/Tomcat/Catalina/Out Prod/MyApp2/MySQL/SlowQueries Dev/MyApp2/Nginx/Access Dev/MyApp2/Tomcat/Access Dev/MyApp2/Tomcat/Catalina/Out Dev/MyApp2/MySQL/SlowQueries
- 17. Sumo Logic ConfidentialSumo Logic Confidential Metadata: Source Category Best Practices and Benefits Simple Search Scoping _sourceCategory=Prod/MyApp1/Apache* (All Apache Logs for Prod) _sourceCategory=*/MyApp1/Apache* (All Apache Logs for all environments) Simple, Intuitive and Self-maintaining Partitions/Indexes _sourceCategory=Prod/MyApp1* _sourceCategory=Prod/MyApp2* Note: First or first and second component are used for partitioning Simple and Self-maintaining RBAC Roles _sourceCategory=Prod/MyApp1*
- 18. Sumo Logic ConfidentialSumo Logic Confidential Metadata: Source Category Best Practices Common components (and any combination of): • Environment (Prod/UAT/DEV) • Application Name • Geographic Information (East vs West datacenter, office location, etc.) • AWS Region • Business Unit Highest level components should group the data how it is most often searched together: Prod/Web/Apache/Access Dev/Web/Apache/Access Prod/DB/MySQL/Error Dev/DB/MySQL/Error Web/Apache/Access/Prod Web/Apache/Access/Dev DB/MySQL/Error/Prod DB/MySQL/Error/Dev
- 19. Sumo Logic Confidential Optimization Tools
- 20. Sumo Logic Confidential Partitions Indexes for subsets of your data. Segregate your data into smaller, logical chunks, that are mostly searched in isolation of other Partitions. Best Practices No overlap < 20 Partitions Ideally between 1% and 30% of total volume Group data that is searched together most often About Partitions Examples: _sourceCategory=Prod/MyApp1* _sourceCategory=Prod/MyApp2* or _sourceCategory=Prod/* _sourceCategory=Dev/*
- 21. Sumo Logic Confidential Field Extraction Rules Apply parse logic for a dataset at time of ingest, as opposed to at search time. Benefits Better Performance Standardized field names Simplified Searches Best Practices Build simple, specific Rules Test Parse and other operations thoroughly (use nodrop and isEmpty for testing) Limitations 50 rules/200 fields (Will be removed soon) Not all operators supported
- 22. Sumo Logic Confidential Scheduled Views Copies of subsets of data, similar to a relation DB materialized view. Use Cases Pre-aggregated data (e.g. for long-term trends) Find the needle in the haystack…. Best Practices We recommend selectivity of > 1:10000 How They Work View is updated by service ~once a minute Allows for backfilling Search view using _view=[viewname] Data does count against ingest volume
- 23. Sumo Logic Confidential Review: Search Optimization Tools What I want to do is Partition Scheduled View Field Extraction Run queries against a certain set of data Choose if the amount of data is between 1-30% Choose if the amount of data you’d like to segregate is 1% or less Choose if you want to pre-extract fields that you are searching against frequently Extract fields from logs and make available to all users ✔ Use data to identify long- term trends ✔ Segregate data by Metadata ✔ Pre-computed or aggregate data ready to query ✔ Use RBAC to deny or grant access to the data ✔ ✔
- 24. Sumo Logic Confidential In Summary, you can… Ingest any type of logs (structured and non-structured) Select a deployment option that best fits your sources Develop a robust naming convention for your metadata Take advantage of Optimization Tools Call to Action: Set up deployment option or (hybrid option) that best fits your environment Ensure you have a robust _SourceCategory naming convention At the very least, set up Field Extraction Rules for your popular data sources
- 25. Sumo Logic Confidential Questions? Consume Training sumologic.com/training Read Documentation help.sumologic.com Search/Post to Community community.sumologic.com slack.sumologic.com Open a Support Case support.sumologic.com Log a Feature Request sumologic.ideas.aha.io/ideas
- 26. Sumo Logic Confidential Thank you!
Editor's Notes
- #2: Welcome everyone to the ”Setting Up Sumo Logic” webinar My name is …. And I am … Before we get started, let’s cover some housekeeping items: - To avoid distractions, all Participants are muted - However, if you want to ask a question, feel free to use the GoToWebinar Question panel. We will have a Q&A session at the end. - To preempt the most common question: Yes, this session will be recorded and I will share Slides and a recording of this webinar with everyone OK, let’s get started
- #4: Sumo Logic helps you gain insights into the growing pool of data within your complex environment.
- #5: Most of you are using the Sumo Logic service for at least one of the 3 following use cases: For DevOps –allows DevOps teams to monitor KPI’s to deliver quality software; less time troubleshooting and more time developing code. For IT Ops – Extract valuable information such as latencies, performance metrics, trends and any critical events tied with core systems. For Compliance and Security – Sumo Logic helps organizations simplify and automate compliance & security monitoring across their entire stack, using predictive analytics
- #6: What data can we ingest? We can ingest data from just about any source you can imagine - structured or unstructured. Here are just a few of the devices, applications and frameworks you may be using - all of which produce log data that Sumo Logic can ingest and analyze. The left hands side can present you technology stack – from custom application code all the way down to your network devices. The right can represent your infrastructure.
- #8: Sumo Logic Data Flow is broken into 3 main areas: Data Collection through configurable Collectors and Sources. Collectors collect, compress, cache and encrypt the data for secure transfer. Search and Analyze – Users can run searches and correlate events in real-time across the entire application stack. We will be spending most of our time in this area during this webinar, as this is most likely what you will first be doing as a new user. Visualize and Monitor- Users have the ability to create custom dashboards to help you easily monitor your data in real-time. Custom alerts notify you when specific events are identified across your stack. I will cover Data Collection at a high-level, and cover the next 2 areas through a demo.
- #10: Sumo Logic Installed and Hosted Collectors are infinitely flexible. Design a Sumo Logic deployment that's right for your organization. <Review slide citing some examples>
- #11: Hosted Collectors Allow for seamless collection from Amazon S3 buckets and HTTP Sources. Hosted Collectors don't require installation or activation, nor do Hosted Collectors have physical requirements, since they're hosted in AWS. Because there are no performance issues to consider, you can configure as many S3 and HTTP Sources as you'd like for a single Hosted Collector. Installed Collectors Sumo Logic Installed Collectors are lightweight and efficient. You can choose to install a small number of Collectors to minimize maintenance or just because you want to keep your topology simple (Centralized). Alternatively, you can choose to install many Collectors on many machines (Local) to distribute the bandwidth impact across your network. Installed Collectors are deployed in your environment, either on a local machine, a machine in your organization, or even an Amazon Machine Image (AMI). Installed Collectors require a software download and installation. Upgrades to Collector software are released regularly. A few things to consider: Consider having an Installed Collector on a dedicated machine if: You are running a very high-bandwidth network with high logging levels. You want a central collection point for many Sources. Consider having more than one Installed Collector if: You expect the combined number of files coming into one Collector to exceed 500. Your hardware has memory or CPU limitations. You expect combined logging traffic for one Collector to be higher than 15,000 events per second. Your network clusters or regions are geographically separated. You prefer to install many Collectors, for example, one per machine to collect local files. IMPORTANT: For system requirement details, see Installed Collector Requirements.
- #12: The Sumo Logic Collector is installed on all target Hosts and, where possible, sends log data produced on those target Hosts directly to Sumo Logic Backend via https connection.
- #13: The Sumo Logic Collector is installed on a set of dedicated machines, these collect log data from the target Hosts via various remote mechanisms and forward the data to the Sumo Logic Backend. This can be accomplished by either using Sumo Logic syslog source type or by running Syslog Servers (syslog-ng, rsyslog), write to file, and collect from there.
- #16: Great, data is ingested into the Sumo Logic service, but something else is also happening in the background. Every single message ingested gets tagged with metadata that makes it much easier to search for related messages. This table shows the 5 main tags (review them all) In particular, I want to point out the source Category metadata field, as choosing the right naming convention can make a big impact on your searching capabilities and performance.