Power BI Advance Modeling

Editor's Notes

• #2: Purpose of this presentation: Power BI Modelling Target audience: Analysts, Report Builders
• #3: Introduce Sami
• #9: In a good Data Model, the model can assume that column types are homogeneous (as they are strongly typed). This is different from Excel, where any given cell in a spreadsheet can contain different data types.
• #17: You will demo these phases shortly
• #18: Similar to Excel with many V-Lookups Denormalized are flattened versions where all attributes are copied into the same table This is a requirement of a certain “T” competitor – and a major differentiator in the market
• #24: A Relationship is analogous to how an Excel VLOOKUP function brings two tables together Can observe which side is the many (*) and which is the one (1) - CARDINALITY When you draw the relationship, Desktop does several heuristic things It checks that one side of the relationship is the “One” and the other is the “Many” Directions of Relationship Uni-directional are created by default and allows filters to get passed from attribute (dim) table to the Fact Bi-Directional relationship allow you to pass filters in both directions This is different than Many to Many There is a significant performance penalty for Bi-Directional filtering
• #25: This build slide emphasizes that the Query editor is for transformations and is a different window than the primary modeling views.
• #26: Quick note with 2 pictures Black Nav bar that shows a live connect doesn’t have the relationship or the data tab Direct Query doesn’t have the Relationship button
• #29: Combining DirectQuery and import in a single Power BI dataset is supported (composite modeling).   A few things to consider: Removing fields—best practice is that if it isn’t a field you’re reporting on, think very carefully about whether it needs to be included in the model Reducing the number of text fields with many unique values—see this whitepaper (https://msdn.microsoft.com/en-us/library/dn393915.aspx) for more information on how tabular models compress data and what you can do to identify the worst offending fields and optimize compression or performance Reducing the date range of data being brought in—it is common to see business users want to bring in all historical data—it’s worth asking whether their business has fundamentally changed in the last X years where trends from before X years are not relevant
• #31: It’s worth mentioning that the product team recommends DQ mode only in situations where low latency is the driving factor. If you have a situation where import is resulting in too large of a dataset, you should consider optimizing the data you are pulling in to better take advantage of the tabular model’s in-memory columnar datastore that’s running in Power BI’s back end.
• #32: No Additional Notes.
• #33: Every Row is stored in a separate “File” How many files do I open to get the “total Sales”? For ANALYTICAL - To get the total sales – it will only need to open ONE File * Column based architecture is better for Analysis
• #34: We tend to be data “Hogs” - which columns do you really need? In a proper Star Schema, Dictionary Encoding would primary affect Dimension table, or text fields on Fact table (of which there should be few, if any).
• #35: Image compression technique which is used to compress black and white images. The PowerBI team is using this same Run Length compression. If you have 10,00 TRUEs then 10,000 FALSEs, it will provide the best compression of the column Memory compression and performance are directly related Mention Kaspers’ calculation blog post to find out how your memory is used. (Link in Post)
• #36: * When the file is closed – it is saved to the hard disc When opened, the DB is stored in RAM (Approx. 60% of machine’s available RAM) Hard Disk has mechanical parts - read/writes are slower In-Memory – read/writes are faster RAM is precious, and the PowerBI team uses many techniques to compress data In the Service, datasets are limited to 1 GB.
• #37: * When the file is closed – it is saved to the hard disc When opened, the DB is stored in RAM (Approx. 60% of machine’s available RAM) Hard Disk has mechanical parts - read/writes are slower In-Memory – read/writes are faster RAM is precious, and the PowerBI team uses many techniques to compress data In the Service, datasets are limited to 1 GB.
• #38: Data Types are how data is stored, as opposed to Data Formats which are how data are displayed Discuss the difference between Fixed Decimal (Currency) and Decimal (which has floating decimal) Fixed Decimal is 19.4 – 19 places to left of decimal and 4 to the right, which allows for proper processing of pennies. Best for Currency numbers. Decimal can be less efficient when the data has excessive precision Discuss splitting DateTime into Date and Time columns so that they compress better Fixed Decimal numbers – stored like whole number uses less memory than Decimal
• #39: Can approximate hierarchies in the drill path of a visual
• #40: Sort by Column was shown in DIAD, but it is important modeling concept to know. This is applied in the Data section of the model.
• #41: This step applies specifically to Q&A (and not to Power BI reports in general). Users often have a variety of terms they use to refer to the same thing, such as total sales, net sales, total net sales. Power BI’s model allows these synonyms to be added to tables and columns within the model. This step can be important. Even with straightforward table and column names, users of Q&A ask questions using the vocabulary that first comes to them, and are not choosing from a predefined list of columns. The more sensible synonyms you can add, the better your users' experience will be with your report. To add Synonyms, in Relationships view, select the Synonyms button in the ribbon, as shown in the following image. Be careful when adding synonyms, since adding the same synonym to more than one column or table will introduce ambiguity. Q&A utilizes context where possible to choose between ambiguous synonyms, but not all questions have sufficient context. For example, when your user asks “count the customers”, if you have three things with the synonym “customer” in your model, they might not get the answer they are looking for. In these cases, make sure the primary synonym is unique, as that is what is used in the restatement. It can alert the user to the ambiguity (for example, a restatement of “show the number of archived customer records”), hinting they might want to ask it differently.
• #44: This slide emphasis learnings from Slide 26-28 – importance of compression
• #45: This slide emphasis learnings from Slide 26-28 – importance of compression
• #46: This slide emphasis learnings from Slide 26-28 – importance of compression
• #47: This slide emphasis learnings from Slide 26-28 – importance of compression
• #48: This slide emphasis learnings from Slide 26-28 – importance of compression Default Segment Sizes: AS Azure: 8 million SSAS: 8 million Premium: 1 million Shared: 1 million PowerPivot: 1 million
• #49: This slide emphasis learnings from Slide 12– use bi-directional filtering with caution
• #50: Fields like Year, which are numeric, but you do not want to summarize, set the global Default Summarization on the Modeling Ribbon We will deep dive into default summarization in Slide 62
• #51: Companies build a mega model to answer all questions. If you build one mega model – model becomes extremely large impacting performance (both in shared and premium capacity). In this scenario – might be good idea to break down the model into smaller models. Maybe Execs use smaller subset or aggregated data. So it maybe be a good idea to break model based on consumers and have multiple models answering different questions at different levels. Not covered here – In Power BI Service maybe good idea to have a Premium workspace for Execs and different workspace for others.
• #54: Even though this class is 200 this is really a 100 level DAX class DAX answers are not “one size fits all” – we are trying to help you find some hidden gems
• #55: Calculated Columns and Measures are both written in the DAX Language. A Calculated Column is evaluated as a new column in the table in which it resides and will not change value until the underlying data is refreshed. Measures are calculations which do not have a result until they are used in a visualization. They may use sums, averages, minimum or maximum values, counts, or more advanced calculations; and they change value in response to your interaction with your reports. A measure needs to be defined in a table but does not really belong to the table and can be moved from one to another without losing its functionality. Differences between calculated columns and measures – even if they look similar, there is a big difference between these. The value of a Calculated Column is computed during data refresh and uses the current row as a context; it does not depend on user activity on the pivot table. A measure operates on aggregations of data defined by the current context. A measure always operates on aggregations of data under the evaluation context.
• #56: Values use either an implicit Measure or Explicit Measure (explicitly written DAX) Use Demo 5:
• #57: Note to Instructor: This has already been covered and is just reinforcing a concept covered in a previous slide.
• #58: Calculated Columns and Measures are both written in the DAX Language. A Calculated Column is evaluated as a new column in the table in which it resides and will not change value until the underlying data is refreshed. Demo 3: In Product Dim Table add the above formula from the Tab called “Calculated Column”
• #59: DEMO 3: If a question comes up related to how to create a calculated column in ”M”, then do the following: Open Query editor and create a calculated column using this formula: if [Unit Price] <= 25 then "Low" else if [Unit Price] <=50 then "Medium" else "High“ M is case sensitive and the if/then/else syntax is different than Excel/DAX Instructor Talking Point, if needed: Once the column comes in it becomes like any other column in Power BI Desktop file The column is compressed There is a secondary processing for Calculated Columns – Which makes process times slower
• #61: First Bring in the column. Let it resolve and then multiply Demo 4: In SalesFact Table write this formula: SalesCOGS = RELATED(ProductDim[Unit Cost]) * Sales[Units] (also available in Calculated Column Tab) Note to instructor: Just like Vlookup but without all of the drama – No NA()
• #62: This is an example of “Just because you can, does not mean you should” Creating a calculated column like this on the fact table takes way too much memory when it could just as easily be placed on the GeographyDim table. No Demo on this slide Note to Instructor: If you need to aggregate back from 1 side to the Many side, there is a companion formula called RelatedTable – but we will cover that later.
• #63: Instructor Talking Point, Best Practice is to create a calculated column at the HIGHEST level of granularity - in the highest dim
• #64: Instructor Note: From the Definitive Guide to DAX (Russo, Marco and Ferrari, Alberto) Calculated Column is just like any other column in a table and you can use it in rows, columns, filters, or values of a pivot table or any other report. You can use it to define a relationship, if needed. The DAX expression defined for a calculated column operates in the context of the current row of the table to which it belongs. Any reference to a column returns the value of that column for the current row. You cannot directly access the values of other rows. – Important concept is that a Calculated Column is computed during the processing and then stored into the model. The Calculated Columns occupy space in memory. Measures are another way to define calculations but when you do not want to compute values for each row but, rather, you want to aggregate values from many rows in a table. A measure needs to be defined in a table but does not really belong to the table and can be moved from one to another without losing its functionality. Differences between calculated columns and measures – even if they look similar, there is a big difference between these. The value of a Calculated Column is computed during data refresh and uses the current row as a context; it does not depend on user activity on the pivot table. A measure operates on aggregations of data defined by the current context. A measure always operates on aggregations of data under the evaluation context.
• #66: Measures are calculations which do not have a result until they are used in a visualization. They may use sums, averages, minimum or maximum values, counts, or more advanced calculations; and they change value in response to your interaction with your reports. A measure needs to be defined in a table but does not really belong to the table and can be moved from one to another without losing its functionality. DEMO 5: From the Modeling Ribbon, show measure called Total Sales and that resolves to the same values as default summarization. Formula: Total Sales = SUM(Sales[Sales Amount]) In the Measure tab, drop the Total Sales measure onto the graph on the right in the values section Instructor Talking Points: Explain the calculations in DAX vs Power BI: On the graph on the top left, the calculations are done by Power BI implicitly. The graph on the right, show the calculations on the right in DAX which are written, explicitly. Instructor Note: Talk about Home Tables. Even though it looks like Measures are in a Fact Table internally they are stored in a separate table called Measures Table Always ensure the Home Table is a FACT table Be sure the mention that measures should only be associated with FACT tables, and that measure names must be globally unique within the data model. If class is interested, may be worth mentioning that the reason measure names must be unique is that they end up being associated with a hidden measures-only table that gets created in the data model
• #67: Advantage to writing DAX Simplify DAX by referencing other Measures Quickly reference the list of measures from the formula bar by typing “[“ – As all Measures start with this character
• #68: DEMO 5: Profit Margin = Sales[Total Sales] – Sales[Cogs] Profit Margin % = DIVIDE([Profit Margin], [COGS] )
• #69: DEMO 5: Measure uses functions like ALL and SAMEPERIODLASTYEAR. These functions will be discussed in detail shortly.
• #70: Evaluation Context – Single biggest reason that people give up on DAX Want to Simplify the Evaluation Context
• #71: In Excel, there are SUMIF and SUMIFS functions – Calculate give you that flexibility and much more as it can be used with ANY Aggregation/Calculation MYR – Drill to one Department’s data – and change the shape and it will all still work
• #72: Note that even if you apply a “Device” Slicer to this page, the measure [Desktop Sales] will always show Desktop
• #73: ALL Filters arguments are ANDs – FILTERS ARE CALCULATED FIRST (to filter the rows), then the math is applied
• #74: Transition slide
• #75: Demo 6: Table from “Calculate-Add” report Note the effect that the Year Slicer has on the table (All items change) When the Device Slicer is selected, only “Total Sales” changes. Filter Context : Jan + 2015 Jan + 2015 + Desktop Jan + 2015 + Tablet
• #77: Optionally you can talk about the next three slides and then go into a demo to show all 3 in action instead of demoing each individually. DEMO 6” “Calculate- Ignore” Tab: The table visualization, with each slide, expose the related column Expose measure called [Total Sales All Geo] – Ignore filter on ANY column from the GeographyDim table, but allows filters from Year Discuss that this measure can be used to calculate a % of Total measure
• #78: DEMO 6: “Calculate- Ignore” Tab: The table visualization, with each slide, expose the related column Expose measure called [Total Sales All State] – Ignore filter on the STATE column from the GeographyDim table, but allows filters from Year Slice on City (Aberdeen is good)
• #79: DEMO 6: “Calculate- Ignore” Tab: The table visualization, with each slide, expose the related column Expose measure called [Total Sales All Selected State] – Ignore filter on the STATE column from the GeographyDim table, but allows filters from Year Slice on City (Aberdeen is good), then select multiple State, see that the measure now only displays the total for selected states.
• #81: DEMO 6: “Calculate- Update” Tab: The measure called [2014 Sales] – Ignores filter on the Year Slicer.
• #82: Transition to Evaluation Contexts: Row & Filter
• #85: Two gods you can invoke in your DAX calculation Can be blended in a single calculations
• #86: Filter Context is defined by the Visual
• #87: Discuss the filter context for the selected cell Think in terms of co-ordinates (latitude/longitude)
• #88: Charts and Graphs also have Filter Context. EACH data point has a unique Filter Context
• #89: A measure is not evaluated until it is in a Visual – and will be under a filter context
• #90: In a Uni-Directional relationship, the filter can only flow from the Dim to the Fact In a Bi-Directional relationship, the filter can flow each way.
• #91: In a Uni-Directional relationship, the filter can only flow from the Dim to the Fact In a Bi-Directional relationship, the filter can flow each way.
• #92: Show going to the data model and updating the relationship from Single to Both (or back) Toggle between this and the next slide Why can’t I just enable all bi-directional relationships They are expensive Only turn them from Uni- to Bi- when absolutely necessary
• #93: See Barbara’ Blog on the many to many using key words (also note there is a custom Key Word text filter)
• #94: Row context invoked when you refer to a column in a calculated column formula without an aggregation function surrounding it Row Context – How many units sold in Row 1?
• #95: Walk through demo: Go to ProductDimTable Create a column = SUM(Sales[Sales Amount]) – This will show total sales for the entire data model not taking the row context into account. Show the first Calculated Column, the SUMX and RELATEDTABLE functions transfer row context to filter context. In the second formula wrapping the SUM in a CALCULATE does the same thing.
• #97: Scalar is a fancy word for using a single coordinate in the evaluation What if you want bring in a Table or List of values and apply your calculation to them?
• #98: The Region = “Central” is in a ROW Context as it is not surrounded by aggregate function
• #99: The table could be calculated ALL(DimGeography[State]) or a table in the data model DimGeography
• #100: An iterator function takes two arguments – we will walk through this example over the next couple of slides
• #101: Like a virtual calculated column, but don’t need to perpetually keep the column. Argument 1 – iterate thru each of the columns in the table argument
• #102: Perform the calculation for each row in the table argument
• #103: Sum the results of all the rows. Note: This calculation respects the Filter Context of the visual This example is effectively SumProduct in Excel
• #104: DEMO: In your working copy of “Student Modeling Pre-class.pbix” where you have the calculated column [COGS] – * Save the file and note the size for the class * Remove the [COGS] calculated Column and re-save the file. * Note that the file size is about 8% smaller. While file size is not an exact measure of memory use, it is a proxy that most students understand.
• #105: Tab: Time Intelligence-YTD Slides walk thru the logic – Follow the narrative of the next 5 slides
• #112: Additional calculations to discuss commonly used scenarios Calculates based on same day in prior month MoM – Month over Month
• #115: By using a variable, you can get the same outcome, but in a more readable way. In addition, the result of the expression is stored in the variable upon declaration. It doesn’t have to be recalculated each time it is used, as it would without using a variable. This can improve the measure’s performance. https://docs.microsoft.com/en-us/dax/var-dax
• #116: Emphasizing what is covered in Slide 66
• #117: Typically data modelers convert blanks to zeros: While loading data in Power Query In Measures to display 0 instead of blanks slowing down the measure
• #118: Usage example Measure= IFERROR([Measure 1] / [Measure 2], 0, [Measure 1] / [Measure 2]) Or in a calculate column: =IFERROR(SEARCH(“RP*”,DimProduct[Product]),-1) If DAX measure has multiple steps and there is a check for error. E.g. If there is an error do X else do Y. Logically this is correct but slows down the measure
• #119: https://docs.microsoft.com/en-us/power-bi/desktop-performance-analyzer
• #120: External Support Resources for Microsoft