Grab: Building a Healthy Elasticsearch Ecosystem
0 likes964 views
The document discusses how Grab uses Elasticsearch (ES) and cloud technologies to address diverse use cases across different teams, highlighting operational challenges like search and monitoring in a complex environment. It emphasizes the importance of customized solutions for various user groups, ensuring access control based on user demographics and roles. Additionally, the evolution of Grab's infrastructure from basic operations to a more integrated and scalable ES solution is outlined, showcasing enhanced performance and management through cloud technologies.
Grab: Building a Healthy Elasticsearch Ecosystem
- 1. Building a Healthy Elasticsearch Ecosystem —— How Grab use ES and cloud technologies among teams with different use cases 1
- 2. • About Grab • Beyond Logs • Among Tech Family • Inside Cloud
- 5. Beyond Logs —— Solving challenging use cases with ES
- 6. Confidential - For Internal Use Only -Confidential - For Internal Use Only - > 15 Online Service ~ 10 Tech Family Multiple challenging use cases Use case types Search Logs Monitoring Support > 24 Prod Cluster
- 7. 7 Solving interesting real life problems with Elasticsearch
- 8. 8 Our Challenges CBD area there are a lot of one-way lanes Pickup / dropoff is usually single lane Point of Interest Problem
- 9. 9 Our Challenges Limited time for waiting and high cost if we miss it Point of Interest Problem CBD area there are a lot of one-way lanes Pickup / dropoff is usually single lane
- 10. 10 Point of Interest Problem ES Solution Flatten ES data as much as possible. Avoid nested query Store hierarchy in relational database Adding cache to decrease latency for search Our Challenges Limited time for waiting and high cost if we miss it CBD area there are a lot of one-way lanes Pickup / dropoff is usually single lane
- 11. 11 Food Merchant Operational Hours Challenge Merchant has different operational hours Our Challenges
- 12. 12 Food Merchant Operation Time Challenge Multiple ops-hours for different meal times Ops-hours and service might be different everyday Our Challenges Merchant has different operational hours (ops-hours)
- 13. 13 Food Merchant Operation Time Challenge Time Fragment, search fragment with term query Range data type and use range query. ES Solution Multiple ops-hours for different meal times Ops-hours and service might be different everyday Our Challenges Merchant has different operational hours (ops-hours)
- 14. 14 Food Merchant Operation Time Challenge Delivery radius varies for different merchants Jakarta Terrain impact the delivery difficulty Sagaing Geoshapes datatype，geoshape search Time Fragment, search fragment with term query Range data type and use range query. ES Solution Multiple ops-hours for different meal times Ops-hours and service might be different everyday Our Challenges Merchant has different operational hours (ops-hours)
- 15. Among Tech Family —— Connect different teams with ES
- 16. Confidential - For Internal Use Only -Confidential - For Internal Use Only - > 15 Services ~ 10 Tech Families User Group Demographic Engineer DA/DS Ops CE > 6 User Groups Solving search problem in different case Different production requirement Customised solution for different services Access control between department Deal with different user pain point. Consider different learning curve Provide relational data for DA / DS Automation for Ops team Grab ES Summary
- 17. 17 Fraud Engineer —— Online fraud detection —— They have huge amount of data —— They focus on searching in a large portion of data —— They would require scalability from ES. ENGINEER Service Engineer —— They mainly handle online service. —— Usually they have strong technical background —— The focus on stability, observability, log management for troubleshoot. —— The service QPS always grow with business. Our Customers
- 18. 18 Our Customers CE Customer Service —— Real-time user history inquiry —— Easy and user friendly tools —— Low QPS but latency sensitive
- 19. 19 OPS DBA —— Troubleshooting by searching logs —— Alert / Monitoring other database —— Internal tool usage SRE —— Deploy timeline to track last changes —— Storing application log Our Customers
- 20. 20 DATA ANALYST Product —— Historical data searching Data analyst —— Daily report from data warehouse —— Very complex aggregation and query Our Customers
- 21. 21 INFO SEC User activities —— Audit internal user activity —— Audit user DB access —— Audit query executed on the database Our Customers
- 22. 22 DATA ENGINEER Production -> Data Warehouse —— Data transform and clean up —— Deal with complex SQL from DA, PA. —— Daily report generated from data lake. Our Customers
- 23. Cluster Health? How to protect from bad read —— Reads will affect each other. —— Bad query is not controllable and can not be throttled How to manage read —— Separate critical / non-critical read. Access Control? Security concern —— User should only access their cluster. —— Separate service user and individual user management. What’s the concern? Manage user group —— Different users should see different content. —— Access will be defined based on user group
- 24. Data stratification Hot-cold layer (Read Separate) —— Cold layer -> offline query / aggregation. —— Downstream pipeline from hot to cold layer. API service (Read Priority) —— Micro service help other engineer focus on business logic. Our Solution
- 25. —— Role base access control Our Solution Access Category Issue different VPNs —— Control the amount of people to access the subnet. Jumpcloud —— Integrate individual user access with company user group. —— Automate the user access application process
- 26. 26 Service How do we solve all these problem step by step?
- 27. 27 Service Stability: —— Load balancer to keep up time. —— ECE container service to keep node up 24hrs. Monitoring: —— Shared Monitoring: Cloudwatch, adjacent monitoring tools for L1 troubleshoot. Divide and conquer
- 28. 28 Eng & CE Security Control: —— Accessing using engineer vpn. —— Apply access by User group. —— Kibana provide easy query UI which reduce difficulty to use. Doorman: —— Provide temporary cluster access —— Engineer own the access management Divide and conquer
- 29. 29 Ops & SRE Monitoring: —— ECE cluster metric monitoring. —— Ingest to ELK to store service logs Management: —— Automation tools: Protal, deal with cluster creation, index/ mapping management. Divide and conquer
- 30. 30 Analyst Data Warehouse: —— Build down stream data pipeline. —— Data transform on ES data. —— Provide SQL to access data Load Testing: —— Get production data from warehouse for load testing. —— Require InfoSec approval Divide and conquer
- 31. Inside Cloud —— Leveraging cloud technology
- 32. How we evolve
- 33. How we evolve 2017 Open Source stage Heavy operation —— AWS EC2, apply for weeks. —— IP world, login and start the service —— Oncall nightmare, login to recover nodes. Limited cluster —— 1 critical production cluster —— 1 none-critical cluster. —— Several playground clusters.
- 34. How we evolve 2017 Exploring stage 1 Month 1 cluster —— More team are exploring ES. —— Need to explore a manageable solution. AWS ES —— A manageable solution. —— Save oncall’s life with managed infrastructure. —— Several playground clusters. 2018Q1
- 35. How we evolve 2017 ECE small deployment ECE as our next solution —— Manageable private infrastructure —— Use Elasticsearch as what we learn on the official website. —— Learning how to use ECE. Mgrating to ECE —— Data transfer script —— Lots of communication with service team on migration steps. 2018Q1 2018Q3
- 36. How we evolve 2017 ECE Median deployment From 3 clusters to 30 clusters —— Scale ECE to get stable performance —— Upgrade ECE to get newest features. —— Almost 0 downtime for 1 year. —— We learn a lot from Elasticsearch support. Integrate ECE with Grab —— Automate ECE operation —— Integrate ECE with our Portal —— Integrate ECE with Cloud service 2018Q1 2018Q3 2019Q3
- 37. 37 Coupling Useful Cloud Service with ES Enhance —— ES provide invert index, which facilitate adding index to dataset —— Export RDS logs to troubleshoot —— Add cache for ES search layer —— Build ES down stream —— Indexing —— Logs —— Search cache —— Format —— Analysis —— Pipeline Health Data Flow DB cooperation Multi-layer Monitoring
- 38. 38 RDS Database generate slowlog, error logs. Find the problem by seeing this with Cloudwath. However, it’s lack of searching capability and visualization. Relation DB CW Cloud monitoring can collect logs from multiple services and store in one monitoring service. Still, the UI is not good for searching and lack of customized alerting. Cloud Monitoring Lambda Every log will trigger a piece of code, which handle the RDS log to ES or slack. Its depends on event types, then it will decide the way to deal with logs. Serverless ES Pipeline send logs / events to ES. If it’s indexed properly, users should be able to use Kibana to search the logs. Elasticsearch Alerting Watcher alerting did a lot of work to integrate with those popular alerts tools. We can config the actions to send the alerts properly. Alerts Coupling Useful Cloud Service with ES - Example
- 39. Self-service Platform —— Next stage of providing search in Grab
- 40. Thank You 谢谢 Arkun Terima Kasih Khob Chai ขอขอบคุณ Cảm ơn bạn eက#$ဇ&$တင)ပ+တယ)