Microsoft Fabric Introduction

Editor's Notes

• #2: Abstract Microsoft Fabric Microsoft Fabric is the next version of Azure Data Factory, Azure Data Explorer, Azure Synapse Analytics, and Power BI. It brings all of these capabilities together into a single unified analytics platform that goes from the data lake to the business user in a SaaS-like environment. Therefore, the vision of Fabric is to be a one-stop shop for all the analytical needs for every enterprise. Fabric will cover the complete spectrum of services including data movement, data lake, data engineering, data integration and data science, observational analytics, and business intelligence. With Fabric, there is no need to stitch together different services from multiple vendors. Instead, the customer enjoys end-to-end, highly integrated, single offering that is easy to understand, onboard, create and operate.     This is a hugely important new product that I have spent a ton of hours understanding and simplifying into a deck and demo that I will present to you.  This will shortcut your time to upskill on it so you are prepared to answer customer questions.  My presentation comes from the angle that you are in the field and are familiar with Azure Synapse and want to know how this differs. ----------------------- May public preview, Microsoft build GA by end-of-year MVP for GA, incremental updates, release features over time TODO: can you have multiple workspace per capacities. Show diagram with workspaces all pointing to the same warehouse dataflow how do we do snowflake cloning How will CDC work with no data flows how do we talk to customers about waiting for GA for Fabric when they need to do something now, go Lakehouse route in Synapse slide on mounting a dedicated pool Architecture diagrams on how things are done in Fabric / Use Cases This is giving new power to pbi users enterprise solution vs department wide solution slide https://www.jamesserra.com/archive/2022/06/power-bi-as-a-enterprise-solution/ Slide with Synapse missing items Will copilot be added? Highlight no more dedicated pools - use serverless PBI capacities - how to delegate get email about behind the scenes warehouse PBI desktop vs Fabric – what features are not yet in Fabric as modeling is there now Schema drift Failover TODO specialist: all or nothing with users who talks about it? Segment out Synapse - I want to build something now, what product do I use? Don't use dedicated pools, but will have a mounting option. Use ADF instead of Synpase piplelines Missing features in Synapse and what is better Snowflake compete slides link to S3 Pricing Position Synapse today to move seameless into Fabric - migration path Purview not in here What about synapse database templates Demo built? Ability to request a demo/presentation from PM/GBB who have been doing it
• #3: Fluff, but point is I bring real work experience to the session
• #6: Microsoft Fabric combines Data Factory, Synapse Analytics, Data Explorer, and Power BI into a single, unified experience, on the cloud. The open and governed data lakehouse foundation is a cost-effective and performance-optimized fabric for business intelligence, machine kerning, and AI workloads at any scale. It is the foundation for migrating and modernizing existing analytics solutions, whether this be data appliances or traditional data warehouses.   Talk Track for Greenfield/Growing Analytics Customers – Microsoft Fabric’s SaaS environment makes it easier to deploy an entire end-to-end analytics engine from the ground up at an accelerated pace. With the solution’s built-in security and governance capabilities, you can be rest assured your data and insights are protected.   Talk Track for Existing Synapse Customers – Microsoft Fabric is an evolution of Azure Synapse. You will still be able to enjoy the benefits and limitless scale of Synapse in an easier to use SaaS solution while adopting new capabilities that enhance your entire analytics approach. And with the addition of Power BI, you can help democratize the ability to uncover insights and create interactive reports across the organization, helping everyone make more data-driven decisions in their everyday work.   Talk Track for Existing Power BI Customers – With Microsoft Fabric, you’ll be able to access new and powerful data tools and services like Azure Synapse within the same user experience you already enjoy with Power BI. You can unify these tools with your disparate data sources in the same environment to establish a single source of truth for all data, driving the ability for everyone to uncover more accurate and consistent insights than before. And instead of having to worry about security concerns of a patchwork analytics estate, you can be rest assured your data is protected with the built-in security and governance capabilities.
• #8: 5x5 = 5 seconds to signup, 5 minutes to wow
• #12: https://microsoft-my.sharepoint.com/:v:/p/jamesserra/EQegpHEGCrlMkohlSrJbdjQBpfhlYpfEjL515flGy5UW7w?e=50ksgm
• #13: It is not possible to share capacities across tenants
• #14: https://learn.microsoft.com/en-us/fabric/enterprise/licenses PBI v-cores will evolve to Compute Units (group of 8 v-cores)
• #17: Public preview available March 23rd. Switch in admin console to run the functionality on/off completely. It will be off by default until July 1st, when it will be switched on unless they go into the admin console and say “No, I don’t want to have this switched on starting July 1st”. Can control at the tenant and capacity levels. The Microsoft Fabric (Preview) trial includes access to the Fabric product experiences and the resources to create and host Fabric items. The Fabric (Preview) trial lasts until Fabric General Availability (GA), unless canceled. After GA, the Fabric (Preview) trial converts to the GA version and is extended for 60 days.
• #19: https://microsoft-my.sharepoint.com/:v:/p/jamesserra/EQegpHEGCrlMkohlSrJbdjQBpfhlYpfEjL515flGy5UW7w?e=50ksgm
• #20: Starting with OneLake itself. OneLake provides you a single data lake for your entire organization. Users cannot create OneLake storage. OneLake storage (ADLS Gen2) managed by OneLake API is attached to Fabric tenant. When a workspace is created, a folder is created in OneLake storage (ADLS Gen2 behind the scenes) on a customer tenant.
• #21: Everyone is able to contribute to the same lake no matter which engine you use. We are doing a lot a work to optimize our engines to work directly with delta parquet as their native format for tabular data as you can see with T-SQL for Data warehousing and DirectLake mode in Analysis Service for BI.
• #22: Think of OneLake as an abstraction layer. You can mount existing ADLS Gen2 to it. Virtualization across many storage account. Maintains a single namespace. A shortcut is nothing more than a symbolic link which points from one data location to another. Just like you can create shortcuts in Windows or Linux, the data will appear in the shortcut location as if it were physically there. Today, if you have tables in a data warehouse, which you want you want to make available along side other tables or files in a lakehouse, you will need to copy that data out of warehouse. With OneLake, you simply create a shortcut in the lakehouse pointing to the warehouse. The data will appear in your lakehouse as if you had physically copied it. Since you didn’t copy it, when data is updated in the warehouse, changes are automatically reflected in the lakehouse. You can also use shortcuts to consolidate data across workspace and domains without changing the ownership of the data. In this example, the workspace B still owns the data. They still have ultimate control over who can access it and how it stays up to date. Many of you already have existing data lakes stored in ADLS gen2 or in Amazon S3 buckets. These lakes can continue to exist and be managed externally to Fabric. We have extended shortcuts to include lake outside of OneLake and even outside of Azure so that you can virtualize you existing ADLS gen 2 accounts or Amazon S3 buckets into OneLake. All data is mapped to the same unified namespace and can be accessed using the same ADLS gen2 APIs even when it is coming from S3.
• #23: https://blog.fabric.microsoft.com/en-us/blog/using-azure-databricks-with-microsoft-fabric-and-onelake/
• #25: https://microsoft-my.sharepoint.com/:v:/p/jamesserra/EQegpHEGCrlMkohlSrJbdjQBpfhlYpfEjL515flGy5UW7w?e=50ksgm
• #26: The Microsoft Fabric Lakehouse analytics scenario makes it so that data can be ingested into OneLake with shortcuts to other clouds repositories, pipelines, and dataflows in order to allow end-users to leverage other data.   Once that data has been pulled into Microsoft Fabric, users can leverage notebooks to transform that data in OneLake and then store them in Lakehouses with medallion structure.   From there, users can begin to analyze and visualize that data with Power BI using the see-through mode or SQL endpoints.
• #27: If you don’t see a Lakehouse table in the warehouse (default), check the data format. Only the tables in Delta Lake format are available in the warehouse (default). Parquet, CSV, and other formats cannot be queried using the warehouse (default)
• #28: Warehouse mode in the Lakehouse allows a user to transition from the “Lake” view of the Lakehouse (which supports data engineering and Apache Spark) to the “SQL” experiences that a data warehouse would provide, supporting T-SQL. In warehouse mode the user has a subset of SQL commands that can define and query data objects but not manipulate the data. You can perform the following actions in your warehouse(default): • Query the tables that reference data in your Delta Lake folders in the lake. • Create views, inline TVFs, and procedures to encapsulate your semantics and business logic in T-SQL. • Manage permissions on the objects. Warehouse mode is primarily oriented towards designing your warehouse and BI needs and serving data.
• #32: https://microsoft-my.sharepoint.com/:v:/p/jamesserra/EQegpHEGCrlMkohlSrJbdjQBpfhlYpfEjL515flGy5UW7w?e=50ksgm
• #33: The Data Warehouse analytics scenario takes existing sources that are mounted, while pipelines and dataflows can bring in all other data that is needed.   IT teams can then define and store procedures to transform the data, which is stored as Parquet/Delta Lake files in OneLake.   From there, business users can analyze and visualize data with Power BI, again using the see-through mode or SQL endpoints.
• #38: Can more than one capacity be connected to a Datawarehouse, for instance, one to handle data writes and one to handle data reads? Currently, a capacity is assigned to the workspace level and a Data Warehouse is associated to a single workspace. This means all artifacts in the workspace will share the same capacity and all read/write operations will use the same capacity. Does Fabric Data Warehouse support fine grained access control like row-level security, column-level security, dynamic data masking? These security constructs are not available but are planned for the Fabric Data Warehouse and will integrate with Fabric’s universal security model. We already support stats, it’s in the docs. Automatic stats on load and on metadata discovery should land soon. Query plans, indexes, SQL RLS etc also will land incrementally
• #41: Python, R, Scala
• #42: https://learn.microsoft.com/en-us/fabric/get-started/decision-guide-warehouse-lakehouse Lakehouse: call it delta lake, owned and managed by Spark, customer can update files - it's user owned. Use if customer likes Spark and using files Warehouse: well structured, SQL front door, transactional guarentees, multi-table transactions. Nobody except the SQL engine can update the files. Use if customer is comfortable with SQL, comes from a relational database world LDF and MDF are still used behind the scenes. Can query the warehouse from the lakehouse, but can’t do opposite. Data is synced into onelake from LDF and MDF (only INSERT works for now) Can't support everything SQL supports with the current open source format at Delta, (multi-table transactions, indexing), so have to use SQL engine Want to get to a point where don't use LDF/MDF files but use delta underneath. Talk about using Iceberg way down the road Other knobs to turn? What should we expose? Hide things (DMV, explain plan) because can't tune. Performance DBA's - getting rid of another part of your job
• #44: Fabric: Can land in bronze zone in warehouse and just use that for all layers to use T-SQL to write Compared to Synapse: no longer having relational storage and dedicated compute - idea is it’s all done within lake 3 deployment models How to organize workspaces – dev/test/prod, by orgs, by cost High concurrency clusters, spark monitoring
• #45: https://microsoft-my.sharepoint.com/:v:/p/jamesserra/EQegpHEGCrlMkohlSrJbdjQBpfhlYpfEjL515flGy5UW7w?e=50ksgm
• #46: Mapping data flows = ADF Data flows Wrangling data flows = Power Query
• #47: Mapping data flows = ADF Data flows Wrangling data flows = Power Query
• #48: Just in the Fabric context, I would say it this way: Fabric Dataflows are the PQ UI with the scale of ADF the Gen1 vs. Gen2 is just the distinction between what is in PBI and Excel today vs. what is in Fabric what will be the response to customers who ask how they move ADF Data flows to Fabric? We are looking at ways to convert to Fabric data flows and work with Partners who can help with the conversions like 1 Since ADF PQ already had cloud scale, why not just move that into Fabric dataflow gen2? so Fabric Dataflow Gen2 will eventually be an improvement over ADF PQ
• #49: https://microsoft-my.sharepoint.com/:v:/p/jamesserra/EQegpHEGCrlMkohlSrJbdjQBpfhlYpfEjL515flGy5UW7w?e=50ksgm
• #50: DirectLake mode is a groundbreaking new engine capability to analyze very large datasets in Power BI. The technology is based on the idea to load parquet-formatted files directly from a data lake without having to query a Data Warehouse or Lakehouse endpoint and without having to import or duplicate data into a Power BI dataset. DirectLake is a fast path to load the data from the lake straight into the Power BI engine, ready for analysis. It loads the data directly from the files into memory at runtime. Because there is no explicit import process, it is possible to pick up any changes at the source as they occur, thus combining the advantages of DirectQuery and import mode while avoiding their disadvantages. DirectLake can read parquet-formatted delta files, but for best performance you should compress the data using the VORDER compression method (50%-70% more compression).
• #51: https://supportability.visualstudio.com/Power%20BI/_wiki/wikis/PBICSSWIKI/786636/Readiness-Direct-Lake-(SeeThru)-mode-in-PBI-reports
• #52: Polaris engine is base of Fabric Microsoft Fabric
• #53: Polaris engine is base of Fabric Microsoft Fabric