Deep Dive Into Elasticsearch: Establish A Powerful Log Analysis System With Elastic Stack | On Premises vs SaaS Elastic Stack Comparisons.
1 like1,104 views
The document presents a comprehensive overview of establishing a powerful log analysis system using the Elastic Stack, comparing on-premises and SaaS options, and addressing logistical challenges in handling high data volumes. It introduces key components like Elasticsearch, Logstash, Kibana, and Beats, emphasizing their roles in data ingestion and visualization. The document also highlights the pros and cons of different service models and concludes by stressing the need to choose suitable architecture and budget for optimized performance.
Deep Dive Into Elasticsearch: Establish A Powerful Log Analysis System With Elastic Stack | On Premises vs SaaS Elastic Stack Comparisons.
- 1. DEEP DIVE INTO ELASTICSEARCH Establish A Powerful Log Analysis System With Elastic Stack. On Premises vs SaaS Elastic Stack Comparisons. Tyler DevOps Engineer NFQ Asia Company
- 2. Agenda • Intro. • Overview: Elastic Stack. • Establish a powerful log analysis system with Elastic Stack. • Elastic stack options from cloud providers. • Which one would be fit for us? • Cost Reflections. • In conclusion.
- 3. About Me • In tech for 7+ years. • Technical Project Coordinator @ AVASO Technology Solutions. • Infrastructure Technical Lead @ Betfair Group PLC. • DevOps Engineer @ NFQ Asia. • Member of Vietnam Elasticsearch Community. • Bash/PowerShell languages. • A dog parent :D
- 4. About NFQ Asia • Member of NFQ Company. • 15+ years’ experience in e-business strategy and software development • 300+ professionals. • 4 countries: offices in Lithuania, Germany, Vietnam, Singapore. • Founded in Vietnam since 2015. • Having organized 5 community events/hackathons in Vietnam.
- 7. What is Elastic Stack? • Formerly known as ELK Stack. • ELK - The acronym for three open source projects: elasticsearch, logstash, and kibana. • Distributed, scalable, and highly available (both on premises or SaaS). • The Elastic Stack is the next evolution of ELK. • Supports the lightweight Beats data shippers from ES v2.1.1.
- 8. Elasticsearch • “You know, for Search” • Free, Open Source. • Search engine based on Lucene. • Near real-time searching, analytics and visualization capabilities. • Sophisticated Restful API.
- 9. Logstash • Open source data collection engine that unifies data from disparate sources, normalizes it and distributes it. • The ingestion workhorse for elasticsearch and more. • Real-time capabilities and pluggable pipeline architecture. • Community-extensible and developer-friendly plugin ecosystem.
- 10. Kibana • Open source analytics and visualization platform designed to work with elasticsearch. • Specialized for large volumes of streaming and real-time data. • No code, no additional infrastructure required. • Easily and quickly understandable through graphic representation.
- 11. Beats Platform • “Data shippers” that are installed on servers as agents. • Either elasticsearch directly or through logstash. • Library written based on Golang. • Supports create your own beat for specific use cases.
- 12. ESTABLISH A POWERFUL LOG ANALYSIS SYSTEM WITH ELASTIC STACK
- 13. Rationale • What is log? • How do we solve the production issue as usual? • How much time do you spend investigating the production issue? • Where are the archived log? • Visualization and dashboards?
- 14. The Challenge How do you satisfy the search needs of the application system’s over 2,000 docs per second while simultaneously providing tactical operational insights that help both Development Team and Operation Team iteratively improve the customer experience?
- 15. The Simple Log Analysis Diagram
- 16. Demonstration
- 17. Scalability Rationale • High availability. • Petabyte-scale data is written and/or read frequently. • High scalability. • Sufficient data allocation. • Costs.
- 18. The Elasticsearch Hot-Warm Architecture
- 19. The Elasticsearch Hot-Warm Architecture (cont.)
- 20. ON PREMISES VS SaaS ELASTIC STACK COMPARISONS
- 21. WHAT IS AWS ELASTICSEARCH SERVICE? • Managed service in AWS Cloud. • Introduced in Oct 2015. • Fully managed; Zero admin. • Highly available and reliable. • Built-in Kibana support. • Integrated with other services in AWS ecosystem.
- 23. What is Elastic Cloud? • Launched in Oct 2015. • Provided by Elastic. • High provisioning and scaling. • Hosted in the Cloud Providers. • Service-oriented architecture. • Containerization using Docker. • Fully supports custom plugins and API.
- 25. HOW DO I KNOW WHICH ONE IS FIT WITH ME?
- 26. Specifications comparison sheet Self-managed Elastic Stack AWS Elasticsearch Service Elastic Cloud Enterprise Pros More options and features. Complete control settings and capacity. Access to other APIs Comprehensive ES monitoring solutions. Lowest costs. SaaS. Simplify the operations via APIs. Security by IAM. Automated snapshots*. Encryption at rest. Monitoring included*. Technical supported. SaaS. Fully control through APIs. Technical Supported. Uptime SLA. Feature-rich and complete monitoring product. Available on Marketplace. Cons Self maintenance. Infrastructure matters. No technical supported. X-Pack limit features. Limited control. Less capacity and scalability. Backup once time per day. No plugins, no logs. Medium expensive. Only support I2 series EC2 instances. Most expensive. Imperfect for AWS-hosted solutions.
- 28. Costs Comparison Chart 8,400.38 10,678.56 11,512.51 75,303.17 81,375.17 11,316.98 14,500.26 25,201.1525,201.15 32,035.68 34,537.54 203,318.55 219,712.95 28,319.95 38,295.63 50,402.30 0 50,000 100,000 150,000 200,000 250,000 Elastic Stack (AWS) Elastic Stack (GCP) AWS Elasticsearch Services Elastic Cloud (GCP) Elastic Cloud (AWS) Cost($) Service Models One Year One Year (All Upfront) Three Years Three Years (All Upfront) *Costs calculated based on 3TB-data cluster in multi-AZ in Frankfurt region
- 29. In Conclusion • Elasticsearch leverage the power of analysis ability for both Dev/Ops teams. • Easily operate/maintain the huge cluster of servers and microservices. • Choose the proper architecture depend on application/system. • Estimate the budget to meet the requirements. • Optimize the aggregation to adopt the resources. • High availability oriented system.
- 30. We are hiring… • Java Senior/Lead Developer • PHP Senior Developer • PHP Technical Lead • Front-end Senior Developer • Front-end Technical Lead • Technical Project Manager Simply send us an email with your enclosed updated CV to: career@nfq.asia
- 31. Contact Me LinkedIn: linkedin.com/in/tylernguyen91 Email: tai.nguyen@nfq.asia Telegram: @tylern91