Webinar: Designing a schema for a Data Warehouse
0 likes51 views
The document discusses the importance of a data warehouse (DWH) in integrating scattered company data from various sources for analytical purposes. It outlines the principles of designing a DWH, focusing on concepts like star schema, dimensions, and facts, which are essential for organizing data efficiently for analysis. The document emphasizes that building a DWH requires both technical skills and business knowledge, and briefly mentions complex concepts such as constellation schemas and potential challenges in representation.
Webinar: Designing a schema for a Data Warehouse
- 1. Designing a schema for a Data Warehouse
- 3. DWH
- 4. A company data is scattered over: ● Different databases ● Internal applications ● SaaS applications The latter can be accessible via APIs or downloadable files Why a DWH
- 5. This means: ● Different wire protocols, query languages ● Different schemas, methodically UNdocumented ● Designed to retrieve a single row, not aggregations ● On a technology designed for OLTP ● Conflicting / redundant / incorrect / missing data ● Business metrics are mixed with PII Why a DWH
- 6. Instead, you want data analysts to… ● Connect to a signle SQL database ● With a well-known, standard schema ● Designed for analytical queries ● On a technology designed to run analytical queries Why a DWH
- 7. This standard schema is designed for analytics queries: ● JOIN ● WHERE ● GROUP BY Why a DWH
- 8. Why a DWH
- 9. It's called a Star Schema. Its most basic concepts are: ● A star represents an event: customer buys product ● A dimension is any event characteristic we might use for filtering and ordering: purchase date, delivery date, product name, product category, customer city… ● The grain defined how specific dimensions are: date or month? city or postcode? ● Facts are the measurements we take: cost, discount, number or product bought, etc Why a DWH
- 10. DWH design
- 11. Designing a DWH is a technological activity that requires some business knowledge How to design
- 15. Designing a DWH is a business activity that requires technical skills How to design
- 16. It starts by identifying business processes you want to have more information about Example processes: ● A customer buys a product ● A Google Ads campaign runs ● A courier delivers a pizza How to design
- 17. ● While doing so, write a dictionary of business terms ● Everyone should understand the terms ● In many companies different teams use some terms with different meanings How to design
- 18. Discuss each event with all the people who need information about it Typically people from multiple departments How to design
- 19. Find out: ● Facts - numerical measurements to take (cost, discount, number or product bought, etc) ● Dimensions - Event characteristics that can be used for filtering and grouping: purchase date, delivery date, store, product name, product category, customer city… ● The grain defines how specific dimensions are: date or month? city or postcode? How to design
- 20. Modify the event statement by adding the time and the dimensions that affect its grain ● A customer buys a product ● Customers buy products in a day, in a city ● Customers buy products in a month, in a country How to design
- 21. Dimensions
- 22. Dimensions are the criterias that will be used to ● Aggregate ● Filter ● Order the numbers. Dimensions
- 23. Example: ● Average amount spent ● By customers over 40, in 2024, in France ● Aggregated by store_city, date Dimensions
- 24. Table: fact_in_store_purchase Dimensions date country city customer_dob prod_count total_price 2024/01/15 FR Paris 1950/02/02 2 150.00 2024/01/15 FR Paris 1952/02/02 1 999.99 2024/01/15 FR Avignon 1962/10/04 1 22.50 2024/01/16 FR Paris 1978/12/02 2 10.00 2024/01/16 FR Nice 1977/11/09 1 199.50
- 25. With this simplistic design: ● Adding dimensional columns is a pain ● Loading data into the table is harder ● We can't query a dimension alone ● We can't get a list of things that didn't happen Dimensions
- 26. Dimensions
- 27. Dimensions usually look like this: ● Stored in separate table ● Denormalised. Hierarchies are represented by repeating data ● They have an ID that is unique to the DWH and has no meaning ● Human readable information is stored in other columns ● Which are indexed Dimensions
- 28. Table: dim_city Dimensions continent country city local_name language population Europe Italy Rome Roma it 10000 Europe Italy Milan Milano it 20000 Europe Italy Alghero Alghero it 30000 Europe Italy Alghero Alghero ca 30000 Europe Scotland Edinburgh Edinburgh en 12345
- 29. Facts
- 30. A fact table usually contains: ● References / foreign keys to Dimension tables ● One or more numeric columns (facts) Facts
- 31. Table: fact_in_store_purchase Facts date country city customer_dob prod_count total_price 20240115 15 32 19500202 2 150.00 20240115 15 32 19520202 1 999.99 20240115 15 44 19621004 1 22.50 20240116 15 32 19781202 2 10.00 20240116 15 71 19771109 1 199.50
- 32. To join fact to dimensions: SELECT f.*, dt.date, ct.city FROM fact_in_store_purchase f NATURAL JOIN dim_date dt NATURAL JOIN dim_country ct NATURAL JOIN dim_customer cu WHERE dt.date > 20240000 AND dt.week_day BETWEEN 1 AND 5 AND ct.country = 'France' AND cu.dob BETWEEN 19800000 AND 20000000 GROUP BY dt.date, ct.city ORDER BY dt.date, ct.city Facts
- 33. There are 3 types of fact tables: ● Transaction fact tables ○ The company buys products ● Periodic snapshots fact tables ○ Monthly inventory ● Accumulating snapshots fact tables ○ Multi-step: courier delivers pizza Facts
- 34. Factless fact tables are a special type of fact tables. They don't have any fact column. They are boolean facts: an existing row is TRUE, a non-existing row is FALSE. Facts
- 35. Table: fact_customer_care_call Facts date customer_id operator_id 20240201 87612 927 20240201 999111 2250 20240201 8825 822 20240202 19166 1002 20240202 38410 948
- 36. Time Dimensions
- 37. General rules for time dimensions: ● One dimension for date only, without time ● Primary key: an integer id in the form yyyymmdd ● Add columns for any significant information: year, month, month day, week day, workday, leap year… Facts
- 38. Store day time in a separate column, if needed ● Primary key: integer id in the format hhmm ● Add separate columns for hours, minutes and any other information you might need ● Depending on your needs, add a row for every minute, or hour in the working hours, or half an hour, etc Facts
- 40. ● You typically have multiple star schemas linked together (Constellation Schema) ● Most dimensions should be shared across multiple stars (Conformed Dimensions) ● Two stars might represent the same data with different granularity, so some facts are present in multiple tables ● Make sure that facts are names consistently across stars (Conformed Facts) Constellation schemas
- 41. But DWH is a complex matter…
- 42. We left out many topics, for example… ● How to represent invoice or bill of lading dimensions (1 invoice contains multiple items) ● How to represent dimensions that change over time ● Role playing dimensions and other dimension types ● Data marts, data lakes ● DWH to feed Machine Learning ● …and more Interested? Contact us for a training! What we left out