Level 3 Certification: Setting up Sumo Logic - Oct 2018
2 likes270 views
The document outlines the requirements and best practices for obtaining a Sumo Power Admin Level 3 certification, focusing on data collection strategies, deployment options, and metadata design for effective log management. It emphasizes the importance of creating robust naming conventions and utilizing optimization tools to enhance search performance. Additionally, it describes different data collection methods, including local, centralized, and cloud data collection, while providing guidelines for sharing and recommending content within Sumo Logic.
![Sumo Logic Confidential
Scheduled Views
Copies of subsets of data, similar to a relation DB materialized view.
Use Cases
● Pre-aggregated data (e.g. for long-term trends)
● Find the needle in the haystack
Best Practices
● We recommend selectivity of > 1:10,000
How They Work
● View is updated by service ~once a minute
● Allows for backfilling
● Search view using _view=[viewname]
● Data does count against ingest volume](https://image.slidesharecdn.com/sumojampoweradminlevel3illuminate-180928140115/85/Level-3-Certification-Setting-up-Sumo-Logic-Oct-2018-26-320.jpg)
Level 3 Certification: Setting up Sumo Logic - Oct 2018
- 1. Become a Sumo Power Admin Level 3 Certification
- 2. Sumo Logic Confidential Become a Sumo Power Admin • Deploy a data collection strategy that best fits your environment • Implement best practices around data collection • Develop a robust naming convention for your metadata • Create, share and recommend Searches and Dashboards • Learn to utilize optimization tools to enhance search performance
- 3. Sumo Logic Confidential Demo: Monitor and Troubleshoot ALERTS notify of a critical event METRICS to identify what’s going on LOGS to identify why it’s happening
- 4. Sumo Logic Confidential High-Level Data Flow
- 5. Sumo Logic Confidential Sumo Logic Data Flow 1 2 3 Data Collection Search & Analyze Visualize & Monitor Operators Charts Collectors Sources Alerts Dashboards
- 6. Sumo Logic Confidential Data Collection Strategy
- 7. Sumo Logic Confidential Enterprise Logs are Everywhere Custom App Code Server / OS Virtual Databases Network Open Source Middleware Content Delivery IaaS, PaaS SaaS Security
- 8. Sumo Logic Confidential Designing Your Deployment ● Sumo Logic Data Collection is infinitely flexible. ● Design a Sumo Logic deployment that’s right for your organization. ● Installed versus Hosted Collectors ➔ Learn more: Set Up Sumo Logic
- 9. Sumo Logic Confidential Collector and Deployment Options Hosted Collectors Installed Collectors Centralized Data Collection Local Data Collection = Collector
- 10. Sumo Logic Confidential Collector Considerations Consider having one Installed Collector on a dedicated machine if: • You are running a very high-bandwidth network with high logging levels. • You want a central collection point for many Sources. Consider having multiple Installed Collectors if: • You expect the combined number of files coming into one Collector to exceed 500. • Your hardware has memory or CPU limitations. • You expect combined logging traffic for one Collector to be higher than 15,000 events per second. • Your network clusters or regions are geographically separated. • You prefer to install many Collectors, for example, one per machine to collect local files. ➔ For system requirement details, see Installed Collector Requirements.
- 11. Sumo Logic Confidential LOCAL Data Collection The Sumo Logic Collector is installed on all target Hosts and, where possible, sends log data produced on those target Hosts directly to Sumo Logic Backend via https connection. Source Types Local Files ▪ Operating Systems, Middleware, Custom Apps, etc. Windows Events ▪ Local Windows Events Docker ▪ Logs and Stats Syslog (dedicated Collector) ▪ Network Devices, Snare, etc Script (dedicated Collector) ▪ Cloud API’s, Database Content, binary data Typical Scenarios Customers with large amounts of (similar) servers, using orchestration/automation, mostly OS and application logs - On Premise Datacenters - Cloud Instances Benefits/Drawbacks + No Hardware Requirement + Automation (Chef/Puppet/Scripting) - Outbound Internet Access Required - Resource Usage on Target
- 12. Sumo Logic Confidential CENTRALIZED Data Collection The Sumo Logic Collector is installed on a set of dedicated machines, these collect log data from the target Hosts via various remote mechanisms and forward the data to the Sumo Logic Backend. This can be accomplished by either using Sumo Logic syslog source type or by running Syslog Servers (syslog-ng, rsyslog), write to file, and collect from there. Source Types Syslog ▪ Operating Systems, Middleware, Custom Applications, etc Windows Events ▪ Remote Windows Events Script ▪ Cloud API’s, Database Content, binary data Typical Scenarios Customers with mostly Windows Environments or existing logging infrastructure (syslog/logstash) - On Premise data centers Benefits/Drawbacks + No Outbound Internet Access + Leverage existing logging Infrastructure - Scale - Dedicated Hardware - Complexity (Failover, syslog rules)
- 13. Sumo Logic Confidential CLOUD Data Collection Most Data is generated in the Cloud and by Cloud Services and is collected via Sumo Logics Cloud Integrations. Source Types S3 Bucket ▪ Any data written to S3 buckets (AWS Audit or other) HTTPS ▪ Lambda Scripts, Akamai, One Login, Log Appender Libraries, etc. Google / O365 ▪ Google API and O365 Typical Scenarios Customers using Cloud infrastructure, while it’s possible to rely on Cloud Data Collection entirely, this is not typical. These source types are normally just part of the overall collection strategies Benefits/Drawbacks + No Software Installation - S3 Latency issues - Https Post Caching Need
- 14. Sumo Logic Confidential Metadata Design
- 15. Sumo Logic Confidential What is Metadata? Metadata tags are associated with each log message that is collected. Values are set through collector and source configuration. Tag Description _collector Name of the collector (defaults to hostname) _sourceHost Hostname of the server (defaults to hostname) _sourceName Name and Path of the log file _source Name of the source this data came through _sourceCategory Can be freely configured. Main metadata tag
- 16. Sumo Logic Confidential Source Category Best Practices Recommended nomenclature for Source Categories From least descriptive to most descriptive ➔ Best Practices: Good Source Category, Bad Source Category Component1/Component2/Component3... Prod/MyApp1/Apache/Access Prod/MyApp1/Apache/Error Prod/MyApp1/CloudTrail Dev/MyApp1/Apache/Access Dev/MyApp1/Apache/Error Dev/MyApp1/CloudTrail Prod/MyApp2/Nginx/Access Prod/MyApp2/Tomcat/Access Prod/MyApp2/Tomcat/Catalina/Out Prod/MyApp2/MySQL/SlowQueries Dev/MyApp2/Nginx/Access Dev/MyApp2/Tomcat/Access Dev/MyApp2/Tomcat/Catalina/Out Dev/MyApp2/MySQL/SlowQueries Note: Not all types of logs need to have the same amount of levels.
- 17. Sumo Logic Confidential Metadata: Source Category Best Practices and Benefits Simple Search Scoping _sourceCategory=Prod/MyApp1/Apache* (All Apache Logs for Prod) _sourceCategory=*/MyApp1/Apache* (All Apache Logs for all environments) Simple, Intuitive and Self-maintaining Partitions/Indexes _sourceCategory=Prod/MyApp1* _sourceCategory=Prod/MyApp2* Note: Not all types of logs need to have the same amount of levels. Simple and Self-maintaining RBAC Roles _sourceCategory=Prod/MyApp1*
- 18. Sumo Logic Confidential Metadata: Source Category Best Practices and Benefits Common components (and any combination of): • Environment (Prod/UAT/DEV) • Application Name • Geographic Information (East vs West datacenter, office location, etc.) • AWS Region • Business Unit Highest level components should group the data how it is most often search together: Prod/Web/Apache/Access Dev/Web/Apache/Access Prod/DB/MySQL/Error Dev/DB/MySQL/Error Web/Apache/Access/Prod Web/Apache/Access/Dev DB/MySQL/Error/Prod DB/MySQL/Error/Dev
- 19. Sumo Logic Confidential Ingesting Metrics Graphite-CompatibleAWS MetricsHost Metrics CollectD Dropwizard StatsD AWS CloudWatch Metrics AWS ECS ✓ Learn More: Setting up Host Metrics ✓ Learn More: Setting up AWS Metrics ✓ Learn More: Setting up Graphite Metrics
- 20. Sumo Logic Confidential Sending Metrics to Sumo Logic Custom Code StatsD Server OS/Container Metrics Library StatsD CollectD = Collector Server/Device/ Container Graphite Graphite1. Host Metrics 2. 3.
- 21. Sumo Logic Confidential Content Sharing
- 22. Sumo Logic Confidential Sharing and Recommending Searches and Dashboards Enable your users by sharing and recommending content that is meaningful to them Share Content • Grant View, Edit, Manage Admin Recommended • Call attention to content
- 23. Sumo Logic Confidential Optimization Tools
- 24. Sumo Logic Confidential Partitions Indexes for subsets of your data. Segregate your data into smaller, logical chunks, that are mostly searched in isolation of other Partitions. Best Practices ● No overlap ● <20 Partitions ● Ideally between 1% and 30% of total volume ● Group data that is searched together most often Examples Or _sourceCategory=Prod/MyApp1* _sourceCategory=Prod/MyApp2* _sourceCategory=Prod/* _sourceCategory=Dev/*
- 25. Sumo Logic Confidential Field Extraction Rules Apply parse logic for a dataset at time of ingest, as opposed to at search time. Benefits ● Better Performance ● Standardized field names ● Simplified Searches Best Practices ● Build simple, specific Rules ● Test Parse and other operations thoroughly (use nodrop and isEmpty for testing) Limitations ● 50 rules/200 fields (Will be removed soon) ● Not all operators supported
- 26. Sumo Logic Confidential Scheduled Views Copies of subsets of data, similar to a relation DB materialized view. Use Cases ● Pre-aggregated data (e.g. for long-term trends) ● Find the needle in the haystack Best Practices ● We recommend selectivity of > 1:10,000 How They Work ● View is updated by service ~once a minute ● Allows for backfilling ● Search view using _view=[viewname] ● Data does count against ingest volume
- 27. Sumo Logic Confidential Review: Search Optimization Tools What I want to do is Partition Scheduled View Field Extraction Run queries against a certain set of data Choose if the amount of data is between 1-30% Choose if the amount of data you’d like to segregate is 1% or less Choose if you want to pre-extract fields that you are searching against frequently Extract fields from logs and make available to all users ✔ Use data to identify long-term trends ✔ Segregate data by Metadata ✔ Pre-computed or aggregate data ready to query ✔ Use RBAC to deny or grant access to the data ✔ ✔
- 28. Sumo Logic Confidential In Summary, you can… ● Ingest any type of logs (structured and non-structured) ● Select a deployment option that best fits your sources ● Develop a robust naming convention for your metadata ● Start sharing and recommending content that is useful to your users ● Take advantage of Optimization Tools ● Call to Action: ○ Set up deployment option or (hybrid option) that best fits your environment ○ Ensure you have a robust _SourceCategory naming convention ○ At the very least, set up Field Extraction Rules for your popular data sources
- 29. Questions?
- 30. Sumo Logic Confidential Sumo Logic Confidential In order to get credit for the exam, In YOUR OWN INSTANCE, go to Certification Tab. • Online Exam • 30 Multiple choice questions • 60-minute time limit • 3 attempts