Dwh lecture 12-dm

Editor's Notes

• #3: There were utitlity companies which goes house by house and collect info like meter reading. Now the data is placed on books, and at a centeral place info is entered in computer. Now address remain same, but the reading changes forever. Now the info become redundant. So if data changes it needs to be reflected at a lot of places. So a solution of the problem was normalization which are based on er modeling. The problem was of the slow joins. The er diagram was turned into tables. Which were joined with other tables to collect the info.
• #5: When things were fine then why we need the DMs. Now look a schema which is in the third normal form. See the next slide Now there are some observations about er diagram. Some questions mentioned above. Now an example from real life. If you go somewhere and you want to know which person is the most important one. Yes, he will be one which has people arround him listening what he is saying. But now can you tell which table is more important? One with largest header size and few rows of record or viceversa. Numerical measurements: e.g. sales data, no of items sold and revenue, the factual data. Descriptive: or dimensional information containing data. So what is the benefit of the simplicity if it may raise more questions at every step.
• #6: So all the previous points take us to the new representation demand. This is explained using graph theory: An ER model can have different shape based on the designer. Every model looks different. The above two graphs are same, but different representation. The left graph is more difficult to understand. So this is the graph isomorphism problem, that you have to tell, which two graphs are same and this is a very computationaly tough problem. So the same prob exists with ER diagram, that models appear different for every problem. So these complexities are taking us towards the need of DM.
• #7: Paradox: conflict. An example is that you went in an hospital and said how was the operation, they said the operation was successful but patiant died. So what is the benefit of such successful operation, which could not save a patients life so a paradox. The problem is complex because of so many tables due to normalization. And in erp system this may be in thousands. The real value of er modeling is when you query a single table or few tables then you will have good performance but in dss we by defualt join many many tables, so performance will suddenly go down this is a paradox.
• #8: So a comparison of er against dm. Er modeling is for oltp and dm for dss. Suppose you have a bike, and you decide that when you make home and decide to load the cement for house making the result is your bike will destroy. But if you do it on a truck it will never have any affect. So the problem is using the right thing for wrong problem. In dss we are concerned with higher level or aggregation, so we will not go on minor details. Er diagrams are different for same problem. But when you make system then all systems will have a lot of variation. But in dss the schema do not change normally. There are smart enviornment which generates sql automatically but they may become in a difficulty while optimizing if the schema always changes. But in DM or star schema, it is very difficult to generate the sql. Er schema changes when business changes, so sql generating tool faces difficulty. But in dss the schema remains constant even with the change of business.
• #9: ER model can be simplified using de-normalization and DM.
• #10: So what is a dm or how we tell about a schema that it is optimized for the dss enviornment. The slide points. So the key point is it is simple, logical and intutive. So if it is easy to understand for programmers, it assures better solution. It has two tables fact and dimensional. Fact tables are large and dimensional tables are small. Fact tables are table which store numerical data i.e how much sale, sale revenew. The dimension table has info about dimension i.e time, geography etc. Keys should be sys generated not the business key, so if the business change key should not need the maintainance.
• #11: Map business analyst representation to relational model Data cubes with dimensions and measures Relational design with tables and 1-M relationships (FKs) Dimensions to dimension tables Measures to fact tables Group fact and dimension tables Grain: most detailed measure values stored
• #12: How fact and dimension table connects? In the form of star topology where fact table is in center. Dm is designed to support the rolap operations, where we can run on the go queries.
• #13: Dimensions have hirarchies. i.e books have fiction and text, but you cannot mix them. So the benefit is decision maker can enter at a point in hirerchy to see the details of other hirarchies.
• #14: The above task can be done by two schemas. Star are simple, either you rotate flip or reposition it wont change, but for snowflake if you do this, you will loose the entire meaning. Star schema represents a complete business process e.g. sales, purchases, inventory etc. For each business process we will have different stars.
• #16: Star schema of the previous slide, and things become simplified. We create the fact tables having real (physical) records, we do not run the joins on run time. This is the reason that in pivot4j we analyze a physical and real star by placing the dimensions of our requirements and mdx generates automatically. Once a star is created it doesn’t matter how you analyze it. suppose there are hundered records in each table and 4 tables are involved in a query which needs a join, now against the join the output returns 40 rows for a specific join query. Now to retrieve these 40 rows we have computed 100x100x100x100 steps. Now if these 40 records are placed in a table (fact table) which has 1000 total rows then in worst case we will achieve the correct output in 1000 steps in star instead of 100000000 steps. So ultimately we will achieve enormous performance.
• #18: When we get star schema, we collapsed the hirarchies and make a single table i.e time is now in a single table means we will avoid the sub tables in the form of pk and fk relations, now the name of a column say city will be used in dimensional table instead of FK, it may result the loss of info i.e every city may have the province name fk but now we will not be able to tell the dependency of cities by just looking the diagram. Its disadvantage is that you cannot tell, which element is subset of which element, and what is the level of element in hirerchy. So loss of information. The benefit is that simple schema with few tables as compare to previously hundreds of tables, another disadvantage is the additional space. The simple example could be on next stage.