How we built this: Data tiering, snapshots, and asynchronous search
1 like729 views
The document outlines advancements in Elasticsearch technologies, notably data tiering, searchable snapshots, and asynchronous search, emphasizing their benefits for cost and performance optimization. It includes forward-looking statements about the company's future operations, highlighting the risks and uncertainties that may impact outcomes. The presentation also details the structural organization of data and how features like searchable snapshots improve efficiency and reduce costs.
How we built this: Data tiering, snapshots, and asynchronous search
- 1. How we built this: data tiering, searchable snapshots, and asynchronous search Jason Tedor Elasticsearch Tech Lead
- 2. This presentation and the accompanying oral presentation contain forward-looking statements, including statements concerning plans for future offerings; the expected strength, performance or benefits of our offerings; and our future operations and expected performance. These forward-looking statements are subject to the safe harbor provisions under the Private Securities Litigation Reform Act of 1995. Our expectations and beliefs in light of currently available information regarding these matters may not materialize. Actual outcomes and results may differ materially from those contemplated by these forward-looking statements due to uncertainties, risks, and changes in circumstances, including, but not limited to those related to: the impact of the COVID-19 pandemic on our business and our customers and partners; our ability to continue to deliver and improve our offerings and successfully develop new offerings, including security-related product offerings and SaaS offerings; customer acceptance and purchase of our existing offerings and new offerings, including the expansion and adoption of our SaaS offerings; our ability to realize value from investments in the business, including R&D investments; our ability to maintain and expand our user and customer base; our international expansion strategy; our ability to successfully execute our go-to-market strategy and expand in our existing markets and into new markets, and our ability to forecast customer retention and expansion; and general market, political, economic and business conditions. Additional risks and uncertainties that could cause actual outcomes and results to differ materially are included in our filings with the Securities and Exchange Commission (the “SEC”), including our Annual Report on Form 10-K for the most recent fiscal year, our quarterly report on Form 10-Q for the most recent fiscal quarter, and any subsequent reports filed with the SEC. SEC filings are available on the Investor Relations section of Elastic’s website at ir.elastic.co and the SEC’s website at www.sec.gov. Any features or functions of services or products referenced in this presentation, or in any presentations, press releases or public statements, which are not currently available or not currently available as a general availability release, may not be delivered on time or at all. The development, release, and timing of any features or functionality described for our products remains at our sole discretion. Customers who purchase our products and services should make the purchase decisions based upon services and product features and functions that are currently available. All statements are made only as of the date of the presentation, and Elastic assumes no obligation to, and does not currently intend to, update any forward-looking statements or statements relating to features or functions of services or products, except as required by law. Forward-Looking Statements
- 3. Data Tiers A simple, integrated approach to optimize for cost and performance
- 5. Daily 3 TB/day Weekly 21 TB/week Monthly 90 TB/month Yearly 1 PB/year Time Series Data
- 6. Only when lawyers ask Search Frequency RarelyOccasionallyConstantly
- 7. Hot tier $$$ Data from recent days Highly relevant Searched constantly
- 8. Warm tier $$$ Data from recent weeks Somewhat relevant data Searched occasionally
- 9. Cold tier $$$ Data from recent months Barely relevant data Searched rarely
- 10. Frozen tier $$$ Data from recent years Irrelevant data Searched when the lawyers ask
- 11. Data tiers store data cost-effectively 1 2 3 4 Hot tier Warm tier Cold tier Frozen Tier
- 12. Data Tiers • A simplified way of managing your data, both content and time series • Currently: use node attributes • Very soon: use node roles data_content, data_hot, data_warm, data_cold (and later, data_frozen) • Then, the Elastic Stack will automatically manage the allocation and relocation of data between tiers at the appropriate phase
- 14. Data Tiers • Lower storage costs • Streamlined operations • Deeper insights and new use cases
- 15. Searchable Snapshots Efficiently search your indices stored in cheap cloud object stores like S3
- 17. Disk Hot|Warm
- 18. S3 Disk Cold
- 19. Cold Searchable Snapshots • Back an index by a snapshot, replicas no longer needed • Data is downloaded and persisted locally • For resilience, Elasticsearch automatically recovers from the snapshot when needed • Deep costs savings without really impacting performance
- 20. Hot Warm Cold Snapshot $$ $/2 Searchable Snapshots $$$
- 21. S3 Disk Frozen
- 22. Doc Values Stored Fields Term Dictionary Term Proximity Normalization Factors Point Values Meta Lookup
- 23. Doc Values Stored Fields Term Dictionary Term Proximity Normalization Factors Point Values Meta Lookup
- 24. S3 Disk Frozen
- 25. S3 Disk Frozen
- 26. Frozen Searchable Snapshots • Make an index snapshot look like a regular index • Incoming searches download the necessary index data • A cache persists this so recently-searched snapshots perform as fast as if local • Costs approaching the cost of S3 to store the data
- 27. Hot Warm Cold $$ $/2 Frozen $ Searchable Snapshots Snapshot $$$
- 28. Searchable Snapshots Cold: Use snapshots instead of replicas Frozen: Search a snapshot stored on S3 (or your favorite cloud object store)
- 29. Asynchronous Search Long-running queries execute in the background
- 30. Data analysis: Async Search Tons of DataSlower (Cheaper) Hardware Show Search Progress
- 31. Asynchronous Search Run potentially long-running queries in the background, allowing you to track their progress and retrieve partial results as they become available
- 32. Data Tiers, Searchable Snapshots, and Asynchronous Search A simple, integrated experience for managing the cost and performance of your data