Logmatic at ElasticSearch November Paris meetup
0 likes1,367 views
- The company started with a data analytics tool called ActivePivot but wanted to build a tool to analyze social media, starting with a NoSQL engine but having performance issues, leading them to Elasticsearch. - While Elasticsearch had good performance, scalability, and analytics capabilities, it did not meet their requirements due to high memory usage and lack of multi-field and metric aggregations. - They built their own analytics plugin for Elasticsearch to add these capabilities, using it to build their Focusmatic product for social media analysis and later their Logmatic product for log analysis.
Logmatic at ElasticSearch November Paris meetup
- 1. OUR STORY WITH ELASTICSEARCH November 2014
- 2. WHERE DO WE COME FROM? • ActivePivot by QuartetFS: – InMemory ultra fast business intelligence tool – Mainly for traders and market risk analysts • One of the biggest french success (& secret) story • We are big data crunchers for a long time now
- 3. HOW DID EVERYTHING START? • Created in may 2012 • We wanted to build the perfect tool to understand the social WEB • We started with a very famous NoSQL engine ! But we quickly had problems (performance, clustering, query/txns overlaps, etc…)
- 4. ELASTICSEARCH 2 YEARS AGO BigData'ready' ' Easy'to'use/manage' ' Performance/Scalability' ' Analy:cal'capabili:es' ' Primary'document'storage?'
- 5. IT DIDN’T MATCH OUR ANALYTICS REQUIREMENTS • FieldData cache – High memory consumption – Memory is expensive on the cloud • No multi-field & multi-metric aggregations – We could not build the product of our dreams • But ElasticSearch is modular – We decided to implement our own analytics plugin
- 6. A SEARCH ENGINE WITH OLAP SKILLS • Support lazy loading of fields • Multi-Fields & Multi-Metric aggregations • Ultra-Fast & Efficient – Usage of columnar storage with primitive types – Sub-seconds queries over tens of millions of elements
- 7. CUSTOM AGGREGATION EXAMPLE The'query' Our'facet' 2'dimensions' 2'metrics' The'result'
- 11. ELASTICSEARCH GETS EVEN BETTER • Release 1.0.0 – February 2014 – First version of the Aggregation Engine – Introduction of doc_values • Release 1.2.0 – May 2014 – global_ordinals / Faster Aggregations • Release 1.4.0 – November 2014 – Improv. Circuit Breakers / Safer Aggregations – Improv. doc_values • Every releases since 1.0.0 – More stability – More aggregation capabilies • We had more time to develop other things!
- 12. INTRODUCING LOGMATIC LOG GOODNESS POWERED BY ELASTICSEARCH • Introduced Logmatic.io in private beta this year – Beginning of 2014: A lot of logs projects around us – Our logs experiment: It was an eye opener! (30 VMs / ~6 apps) – 2 friendly startups tried: they went live – Market Study: 12 projects launched – We faced new challenges and had to build a new product!!
- 13. OUR TYPICAL USER ?
- 14. OUR APPROACH Customer’s applications Our cloud based infrastructure CTO, devops, developers Tomorrow, we’ll even have business people. We’ll tell you more… alerts, reports' queries' TCP'(SSL)' • Centralises & enriches all data • Fully hosted (SaaS) • Advanced analytics • Real-Time • Beautiful dataviz • Rapidfire answers secured connection' And'more…' UDP' HTTP(S)'
- 15. DEMO
- 16. INCOMING FEATURES • Integrated Grok parsing: – Logs shipping should be the only concern – Logs structuration is done totally in the cloud – We extended Grok to simplify issues like date parsing
- 17. INCOMING FEATURES • Integrated Grok parsing: – Logs shipping should be the only concern – Logs structuration is done totally in the cloud – We extended Grok to simplify issues like date parsing • And much more… – Security and limited views – Dimension contexts – Complex metrics / formulas
- 18. WE CONTINUE WITH ELASTICSEARCH • Scalability • Heterogeneity • Query performance • Great analytics • Reactivity of the team
- 19. We’d love to hear from you and answer the questions you might have logmatic.io @logmatic_ hello@logmatic.io