Spark Data Streaming Pipeline
Download as PPTX, PDF1 like343 views
The document discusses the SparkPipe framework, a development tool for .NET developers that orchestrates data processing and storage across cloud and on-premises environments. It addresses the challenges of managing large volumes of diverse data, offering a visual pipeline design for building scalable data workflows. Additionally, it provides support for various Azure services and includes components for healthcare data integration.
Spark Data Streaming Pipeline
- 1. Spark DSM Data Streaming Pipeline ORCHESTRATING DATA STORAGE, PROCESSING, AND MOVEMENT
- 2. Background Today’s data landscape for enterprises continues to grow exponentially in volume, variety, and complexity. Multiple geographic locations, on-premises and cloud Combination of open source, commercial solutions and custom processing code Can be expensive, hard to integrate and maintain. Ever increasing volumes of data (terabytes, petabytes) New ways of processing data (Hadoop, Spark etc.) .NET Developers write large amounts of custom point-solution logic Difficult to maintain and orchestrate Performance bottlenecks
- 3. SparkPipe Framework A development framework to deliver a .NET information production system that co-ordinates all of this data and processing. Familiar technologies for .NET developers including .NET Framework 4.0 Windows Workflow Foundation Task Parallel Library Dataflow Drag and drop business process pipeline modeling Designed for performance to scale across processor cores and servers from the local data center to cloud providers such as Microsoft Azure
- 4. Build Solutions Build data-driven workflows (pipelines) that join, aggregate and transform data sourced from on-premises, cloud-based, and internet data stores. Transform semi-structured, unstructured and structured data from diverse data sources into trusted information. Produce data that can be easily consumed by using business intelligence (BI), analytics tools, and other applications. Set up complex data processing through simple composing.
- 6. Built for “Cloud Scale” Support for Microsoft Azure offerings including: Azure SQL Server HDInsight (HADOOP) Blob, Tables, Queues and ServiceBus Automatically spin-up cloud servers, process data and then shut down to for cost-effective processing.
- 7. Support for Healthcare Out of the box components include: HL7 v2 Clinical Document Architecture EDI 834 PGP Encryption Secure FTP