Sarine's Big Data Journey by Rostislav Aaronov
- 1. Shaping diamonds with Elastic Rostislav Aronov, System Architect, Sarine
- 2. • Rostislav Aronov • System Architect @ Sarine • More then 25 years in SW industry • From coder via DBA, IT, DevOps to architect • Currently leading key projects at Sarine - • Cloud connected planning application • Sarine Profile • Others About Me On Time, On quality, On budget!
- 3. • Public company – Singapore exchange • Market share at manufacturing – more than 80% • Millions of the world’s diamonds are touched by Sarine technology • CUT grading – ALL major labs – NGTC, GIA, AGS, HRD, IGI… • Number of employees – 600 World Wide • Video - https://youtu.be/0PL9BtO-xl4 About Sarine Technologies
- 5. About SarineTechnologies Mining & Rough Trade Manufacturing Gemological Reporting Polish Trade Retail
- 8. About SarineTechnologies Perfect Stone High Clarity, Colour and Cut grades Low Clarity Low Colour Low Cut Sarine ProfileTM
- 9. About SarineTechnologies Sarine Planning application
- 10. • Stand-alone (disconnected) desktop application • No usage statistics • No customer feedback • Support done by screen sharing • Application/Client Security only • Applanders - Escaping Appland! About SarineTechnologies
- 11. Sarine – Cloud connected application Work station (client) Sarine Application Back End Service for Data processing Data Store SQL-RDS/DynamoDB/S3 Elasticsearch Logstash Kibana Lambda Statistic collector
- 12. • Application centric logs – requests and results statistic data • .Net Services activity logs – log4net info and error level logs • Client sense logs – requests round trip statistics • Lambda activity logs – info and errors Sarine – Cloud connected application
- 13. • CEO – Now this is 21st century! • PM –We have valuable feedback • CC – It is easy to help our customers • DevOps –We know how to monitor and maintain the production system • Dev –We have cool technology • And more… Elastic and Sarine - BusinessValue
- 14. • Elasticsearch cluster with 3 nodes • 125 indices (daily, weekly, monthly) • 268 shards • 400 millions docs • 1 terabyte data • And counting… Elastic – How we use it?
- 15. • Shield • Marvel • Watcher • Head • Kopf • CB – XDCR • Cloud – AWS for S3 backups • Curator Elastic – How we use it?
- 16. • DataZone • POC • Architecture and Sizing • First implementation • Team education • Elastic subscription • Fine tuning and best practices recommendation • Production monitoring, 24/7 support • Team education Partners for our Big Data Journey
- 17. • More Sarine projects on elastic • Sarine Profile • Main data store – Couchbase • Main log/history system • Main search system • Main Data visualization system • Automatic tests • Results collection • Data visualization, reports • Main monitoring tool • Alerts based on watchers • Main dashboard Elastic – Is a Strategic Partner
- 18. • Use elastic for real-time data collection, analysis and visualization • Use elastic for super fast search as a part of web and desktop application • Plan index structure (shards, periodic, size) • Plan the cluster size, use at least 3 nodes to avoid split brain • Design index mapping carefully. Use templates, analyzers according to your needs. Default setting are good enough for general case only! • Educate you team, it is new and not intuitive for most people technology • Use partners and consulting at least at the begging • Re-indexing is expensive. Avoid it. Elastic – Recommendations
- 19. Q & A Sarine - Shaping diamonds with Elastic
- 20. • Careers at Sarine • Client Ninja / Developer for PresentationTeam • Full stack Developers for Cloud Services & PresentationTeams • Test Engineer to lead QA in Cloud Services team Sarine - Shaping diamonds with Elastic
- 21. Thank You!
Editor's Notes
- #9: http://love-and-co.com/ultimate-light-performance/ https://www.skjewellery.com/collections/jewellery/allove.html#quality-and-assurance-allove