Turning Raw Data Into Gold With A Data Lakehouse.pptx

Editor's Notes

• #3: Should also note for the purposes of this talk while I am talking about Data Lakehouse/Medallion architecture as concepts everything (especially demos) is really rooted in Databricks
• #4: Data lakehouses are more popular than ever. Microsoft fabric is about to be released into general availability so Microsoft has shown a lot of confidence in this technology. Today I’m going to be mostly talking from a Databricks perspective as they are one of the leading providers but its important to point out that lots of what im talking about, especially medallion architecture can be applied to any data lakehouse. So what is a data lakehouse? Data Lake + Datawarehouse The idea behind the data lakehouse is to take the positives of both while addressing at least some of the negatives
• #5: Volume and Variety: A data lake can accommodate the large amount of data that Big Data, artificial intelligence, and machine learning requires. Data lakes can handle the volume, variety, and velocity of data from various sources being ingested in any format. Speed of Ingest: Format is irrelevant during ingest. Rather than schema-on-write, it uses schema-on-read, which means data doesn’t need to be processed for use until it is needed. Data can be written quickly. Lower Costs: Relative to a data warehouse, a data lake can be significantly less expensive when it comes to storage costs. This allows companies to collect a wider variety of data, including unstructured data such as rich media, sensor data from the Internet of Things (IoT), email, or social media. Greater Accessibility: Data stored in a data lake makes it easy to open copies or subsets of data to be accessed by different users or user groups. Data access can be controlled while companies can provide broader accessibility. Advanced Algorithms: Data lakes let companies conduct complex queries and deep learning algorithms to recognize patterns.
• #6: Complex On-Premises Deployment: Spinning off a data lake in the cloud is a simple process. Deploying a data lake on-premises can be significantly more complex. On-prem solutions such as Hadoop or Splunk are available, but data lakes are built for the cloud. Learning Curve: There is a bit of a learning curve, new tools, and new services with a data lake. This requires either outside help, training, or recruiting team members with data lake skill sets. Migration: If you’re already working with a data warehouse, transitioning to a data lake takes some careful planning of your data strategy to manage your data sets. Depending on your infrastructure, this can be a challenge. Handling Queries: It’s fast and easy to ingest data, but a data lake is not optimized for queries in the way structured and semi-structured data is in a data warehouse. Using best practices for database queries can help, but data retrieval is not as straightforward as it is with a data warehouse. In a data lake, data is processed after it’s been loaded using the Extract, Load, Transform (ELT) process. Governance, semantic consistency, and access controls are required, otherwise, the data lake may turn into a “data swamp” of unusable, raw data.
• #7: Maturity: Data warehouses have been around for over 30 years. These are tried and tested tools/methods for storing data. Maintenance: Data warehouses typically perform efficiently with little maintenance. If you do need to work on your warehouses, there are plenty of IT teams with the skill sets needed. Performance: Since data warehouses use a schema-on-write process, the common underlying data structure makes it easy for the query engine to sift through data and generate results quickly. Usability: Rather than sort through raw data or complex data structures, users find it easy to find what they need. Flexibility: Companies can store data in the cloud using one of the many cloud providers available, or on premise, or a hybrid of the two. Data Marts: Data marts can provide structure and access to specific environments within a single functional data set. For example, you can create data marts for specific products or lines of business.
• #8: Storage Costs: One of the downsides of data warehouses is the cost of storage for large volumes of data. Time: Each business process component must be built to extract value from the data. It can be time-consuming to get data into the data warehouse using an ETL process. For example, if you want to do data analytics on current and historical data, but the source data is not currently in the database, it will have to be processed and added before you can examine it. Limited Source Data: Because of storage costs and the need to process data, organizations often limit what data is captured and stored and what data sources are ingested. In most cases, this leads to storing data for known reporting requirements rather than raw data or data you may need in the future. The Big Data Challenge: A data warehouse is not built for Big Data analysis. Data warehouses are not designed/optimized for the massive amounts of data now being collected by some companies. Complicated Changeover: If you want to add new fields it can be tricky. Data must either be reprocessed or historical data will include blank fields. Potential for Data Distortion: Since data quality relies on pre-load data cleansing, errors in the processing means data can be distorted forever.
• #9: Structured - Can still have that table structure that a data warehouse provides Governed – Who has access to what Familiar – We have SQL, Tables, Views etc Fast – We can deal with massive amounts of data very quickly *as long as it’s set up efficiently Flexible – Can deal with structured and unstructured data
• #10: A lot of the negatives of a traditional data warehouse are solved by the modern cloud data warehouses available today. So why would you want to use a lakehouse? Unified platform – Data Engineering, Data Science, Data Analytics Apache Spark – Designed for huge amounts of data Delta Lake – Storage format – will talk about more in a min AI & Machine Learning – Massive at the moment – Dolly LLM – Mlflow is integrated into databricks – great support for deploying and running production models
• #12: Important thing to take away from this slide is that its distributed. Jobs will be split into tasks and then executed in parallel across different worker nodes and then the results combined.
• #14: Versioned Parquet data files + Transaction log This gives ACID transactions: Atomic, Consistent, Isolated and Durable – What we are used to from Data warehouses that are missing from data lakes
• #17: Validation rules - No nulls - Data is unique - Data is correct type Logical checks - Does an order have an order date - For a country field is that a country and is it spelled correctly Reference data - Swapping country, state codes for full names
• #18: - Monthly, yearly sales tables - A business might want a query to show which customers have spent the most in a certain timeframe - You could aggregate the revenue for different LOB weekly
• #20: At AA we use RAW > BASE > CURATED unless a customer wants to have their own convention. This is to hopefully give a better description of the data at each layer