Msbi Architecture
Download as PPTX, PDF3 likes1,110 views
SSIS is a component of SQL Server that allows for data integration and workflow. It has separate runtime and data flow engines. The runtime engine manages package execution and control flow, while the data flow engine extracts, transforms, and loads data in a parallel, buffered manner for improved performance. SSAS is the analysis component that builds multidimensional cubes from relational data sources for analysis. It uses an OLAP storage model and has components for querying, processing, and caching data and calculations. SSRS is the reporting component that allows users to build interactive, parameterized reports from various data sources and deliver them through a web portal.
Msbi Architecture
- 1. Presented By: Rajinder Singh
- 2. What is MSBI • “This suite is composed of tools which helps in providing best solutions for Business Intelligence Queries. These tools use Visual studio along with SQL server. It empower users to gain access to accurate, up-to-date information for better decision making in an organization. It offers different tools for different processes which are required in Business Intelligence (BI) solutions. • MSBI’s 3 Components:- • SSIS – SQL Server Integration Services – Integration tool. • SSAS – SQL Server Analytical Services -Analysis tool. • SSRS – SQL Server Reporting Services – Reporting tool.
- 3. MSBI Components Flow Diagram
- 4. END USER TOOLS & PERFORMANCE MANAGEMENT APPS Excel PerformancePoint Ser ver BI PLATFORM SQL Server Reporting Services SQL Server Analysis Services SQL Server DBMS SQL Server Integration Services SharePoint Ser ver DELIVERY Reports Dashboards Excel Workbooks Analytic Views Scorecards Plans
- 5. BI Supported Platforms Pervasive Insight Dynamic Development Beyond Relational Enterprise Data Platform Mobile and Desktop OLAP FILE XML RDBMS Entity Data Model Services Query Analysis Reporting Integration Synch Search CloudServer
- 6. Highlights of MSBI Technologies • MSBI solutions are built on an enterprise data platform, fully integrated with the tools you’re using today to manage your IT operations and infrastructure • By leverage that IT infrastructure in place today with SQL Server, you provide your users with the trust in the information the demand, the integration they require, and the insight they need to drive better business decisions. • All done in an environment that is highly scalable and ready to meet the most demanding requirements of thousands of users throughout your enterprise. • Importantly, in a dynamic development environment that your IT department knows and uses today, allowing them the rapidly develop, author, and publish key BI deliverables to the end users, from reports, to OLAP cubes, to analytic models embedded in other applications that drive increased insight and better business decisions. • All on the Microsoft technology platform that you use and trust today.
- 7. SSIS Engine
- 9. SSIS is a component of SQL Server 2005/2008 and is the successor of DTS (Data Transformation Services) which formed part of SQL Server 7.0/2000. It has segregated the Data Flow Engine from the Control Flow Engine or SSIS Runtime Engine; designed to achieve a high degree of parallelism and improve the overall performance. SSIS Runtime Engine – The SSIS runtime engine handles the control flow of a package. It saves the layout of packages, runs packages and provides support for logging, breakpoints, configuration, connections and transactions. The run-time engine is a parallel control flow engine that coordinates the execution of tasks or units of work within SSIS and manages the engine threads that carry out those tasks. The SSIS runtime engine executes the tasks inside a package in an orderly fashion. When the runtime engine encounters a data flow task in a package during execution it creates a data flow pipeline and lets that data flow task run in the pipeline.
- 10. SSIS Data Flow Engine/Pipeline – SSIS Data Flow Engine or Data Flow Pipeline or Transformation pipeline engine manages the flow of data from data sources, through transformations, and on to destination targets. When the Data Flow task executes, the SSIS data flow engine extracts data from one or more data sources, performs any necessary transformations on the extracted data and then delivers the data to one or more destinations. The Data flow engine is buffer oriented architecture (more details will be discussed in a later section), it pulls data from the source and stores it in a buffer (memory structure) and does the transformation in buffer/memory itself instead of processing on a row-by-row basis. The benefit of this in-memory processing is that processing is much faster as there is no need to physically copy/stage the data at each step of the data integration; the data flow engine manipulates data as it is transferred from source to destination.
- 12. The diagram shows a typical Microsoft BI application architecture which has different layers shown from left to right. On the left layer you have source systems or a relational data warehouse, in the middle layer you have the Analysis Services cube pulling data from the source systems and storing it in an Analysis Services cube/OLAP store and on the right layer you have reporting applications which consume the data from the Analysis Services cube/OLAP cube.
- 13. Query Parser The Query Parser has an XMLA listener which accepts requests, parses the request and passes it along to the Query Processor for query execution. Query Processor Upon receiving the validated and parsed query from the Query Parser, the Query Processor prepares an execution plan which dictates how the requested results will be provided from the cube data and the calculations used. The Query Processor caches the calculation results in the formula engine cache (a.k.a Query Processor Cache) so it can be reused across users with the same security permissions on subsequent requests. This summarizes the Query Processor operations: Makes a request for sub cube data from storage engine Translation of request into sub cube data requests Produces result set by doing Bulk calculation of sub cube Cell-by-cell calculations
- 14. Stores calculation results in formula engine cache with varying scope Query scope - cache will not be shared across queries in a session Session scope - cache will be shared across queries in a session Global scope - cache can be shared across sessions if the sessions have the same security roles Storage Engine The Storage Engine responds to the sub cube data (a subset or logical unit of data for querying, caching and data retrieval) request generated by the Query Processor. It first checks if the requested sub cube data is already available in the Storage Engine cache, if yes then it serves it from there. If not then it checks if the aggregation is already available for the request, if yes then it takes the aggregations from the aggregation store and caches it to the Storage Engine cache and also sends it to Query Processor for serving the request. If not then it grabs the detail data, calculates the required aggregations, caches it to the Storage Engine and then sends it to Query Processor for serving the request. This summarizes the Storage Engine operations: Creates Attribute Store (Key store, relationship store, bitmap indexes etc) Creates Hierarchy Store Creates Aggregation Store
- 15. • Storage Engine Cache • Loads data from storage engine cache as queries execute • Clears data from storage engine cache with cleaner thread (in case of memory pressure) or processing of partitions • Aggregation Data • Responds to request with aggregated values in storage • If new then summarizes lower level aggregated values on the fly as needed • Fact Data • Scans MOLAP partitions and partitions segments in parallel • Uses bitmap indexes to scan pages to find requested data
- 16. Data Mining
- 17. • Data mining is described as a process of discover or extracting interesting knowledge from large amounts of data stored in multiple data sources such as file systems, databases, data warehouses…etc. This knowledge contributes a lot of benefits to business strategies, scientific, medical research, governments and individual. • Business data is collected explosively every minute through business transactions and stored in relational database systems. In order to provide insight about the business processes, data warehouse systems have been built to provide analytical reports that help business users to make decisions. • Data is now stored in databases and/or data warehouse systems so should we design a data mining system that decouples or couples with databases and data warehouse systems? This question leads to four possible architectures of a data mining system as follows:
- 18. • No-coupling: in this architecture, data mining system does not utilize any functionality of a database or data warehouse system. A no-coupling data mining system retrieves data from a particular data sources such as file system, processes data using major data mining algorithms and stores results into file system. The no-coupling data mining architecture does not take any advantages of database or data warehouse that is already very efficient in organizing, storing, accessing and retrieving data. The no-coupling architecture is considered a poor architecture for data mining system however it is used for simple data mining processes. • Loose Coupling: in this architecture, data mining system uses database or data warehouse for data retrieval. In loose coupling data mining architecture, data mining system retrieves data from database or data warehouse, processes data using data mining algorithms and stores the result in those systems. This architecture is mainly for memory-based data mining system that does not require high scalability and high performance. • Semi-tight Coupling: in semi-tight coupling data mining architecture, beside linking to database or data warehouse system, data mining system uses several features of database ordata warehouse systems to perform some data mining tasks including sorting, indexing, aggregation…etc. In this architecture, some intermediate result can be stored in database or data warehouse system for better performance.
- 19. • Tight Coupling: in tight coupling data mining architecture, database or data warehouse is treated as an information retrieval component of data mining system using integration. All the features of database or data warehouse are used to perform data mining tasks. This architecture provides system scalability, high performance and integrated information.
- 20. SQL Server Reporting Services(SSRS) What is SSRS? Microsoft SSRS or Business Intelligence SSRS, lets you create very rich reports (Tabular/Graphical/Interactive/free-form) from various data sources with rich data visualization (Charts, Maps, spark lines). All these reports can be viewed via a web browsers. SSRS allows are reports to be exported in various formats (Excel, PDF, word etc)SSRS allows reports to be delivered via emails or dropped to a share location in an automated fashion. SSRS Components: • Report Server • Report Builder • Report Manager
- 21. Types of Reports • Parameterized reports • Linked reports • Snapshot reports • Cached reports • Ad hoc reports • Clickthrough reports • Drilldown reports • Drillthrough reports • Subreports
- 22. Reporting Architecture (c) 2008 MResult Corp | Confidential 22 (BIDS) Visual Studio
- 23. SSAS Report Generation in Excel
- 24. Query Builder Reports • The text-based query builder (default) provides a simple workspace for specifying a query and viewing the results. You can specify multiple Transact-SQL statements, query or command syntax for custom data processing extensions, and queries that are specified as expressions. Because the generic query builder does not preprocess the query and can accommodate any kind of query syntax, it is the default query builder tool for Report Designer. • The graphical query builder provides a richer visual experience. It is used in Visual Studio and in other parts of SQL Server. You can use the graphical query builder if you are not creating expressions or multi- part SQL statements. • To switch to the graphical query builder, toggle the Edit As Text button in the top left corner of the window.
- 25. Query Builder Report Viewer
- 26. Drill Through and Drill Down Reports DrillDown Reports You can organize data in a variety of ways to show the relationship of the general to the detailed. You can put all the data in the report, but set it to be hidden until a user clicks to reveal details; this is a drilldown action. DrillThrough Reports You can display the data in a data region, such as a table or chart, which is nested inside another data region, such as a table or matrix. You can display the data in a subreport that is completely contained within a main report. Or, you can put the detail data in drillthrough reports, separate reports that are displayed when a user clicks a link.