10 Big Data Technologies you Didn't Know About
Download as PPTX, PDF14 likes4,413 views
This document introduces several big data technologies that are less well known than traditional solutions like Hadoop and Spark. It discusses Apache Flink for stream processing, Apache Samza for processing real-time data from Kafka, Google Cloud Dataflow which provides a managed service for batch and stream data processing, and StreamSets Data Collector for collecting and processing data in real-time. It also covers machine learning technologies like TensorFlow for building dataflow graphs, and cognitive computing services from Microsoft. The document aims to think beyond traditional stacks and learn from companies building pipelines at scale.
![Apache Flink: An Example
case class Word (word: String, frequency: Int)
val lines: DataStream[String] = env.fromSocketStream(...)
lines.flatMap {line => line.split(" ")
.map(word => Word(word,1))}
.window(Time.of(5,SECONDS)).every(Time.of(1,SECONDS))
.groupBy("word").sum("frequency")
.print()
val lines: DataSet[String] = env.readTextFile(...)
lines.flatMap {line => line.split(" ")
.map(word => Word(word,1))}
.groupBy("word").sum("frequency")
.print()
DataSet API (batch):
DataStream API (streaming):](https://image.slidesharecdn.com/bigdatatechnologiesyoudidntknowaboutpublic-160331190552/85/10-Big-Data-Technologies-you-Didn-t-Know-About-15-320.jpg)
10 Big Data Technologies you Didn't Know About
- 1. Big Data Technologies You Didn’t Know About
- 2. About Us • Emerging technology firm focused on helping enterprises build breakthrough software solutions • Building software solutions powered by disruptive enterprise software trends -Machine learning and data science -Cyber-security -Enterprise IOT -Powered by Cloud and Mobile • Bringing innovation from startups and academic institutions to the enterprise • Award winning agencies: Inc 500, American Business Awards, International Business Awards
- 3. • Big data technologies you didn’t know about • Apache Flink • Apache Samza • Google Cloud Data Flow • StreamSets • Tensor Flow • Apache NiFi • Druid • LinkedIn WhereHows • Microsoft Cognitive Services Agenda
- 4. Two Goals…
- 5. Think Beyond Traditional Big Data Stacks
- 6. Learn from Companies Building Big Data Pipelines at Scale
- 7. Big Data pipelines in the enterprise
- 8. Areas of a Big Data Pipeline Big Data Pipeline Data Processing Stream Data Ingestion Data transformati ons Cognitive Computing Machine Learning High Performance Data Access
- 10. Technology Stacks You Know
- 11. But You Probably Didn’t Know About….
- 12. Apache Flink • Apache Flink, like Apache Hadoop and Apache Spark, is a community- driven open source framework for distributed Big Data Analytics. • Apache Flink engine exploits data streaming and in-memory processing and iteration operators to improve performance. • Apache Flink has its origins in a research project called Stratosphere of which the idea was conceived in 2008 by professor Volker Markl from the Technische Universität Berlin in Germany. • In German, Flink means agile or swift. Flink joined the Apache incubator in April 2014 and graduated as an Apache Top Level Project (TLP) in December 2014.
- 13. Apache Flink • Declarativity • Query optimization • Efficient parallel in- memory and out-of- core algorithms • Massive scale-out • User Defined Functions • Complex data types • Schema on read • Streaming • Iterations • Advanced Dataflows • General APIs Draws on concepts from MPP Database Technology Draws on concepts from Hadoop MapReduce TechnologyAdd
- 14. Apache Flink
- 15. Apache Flink: An Example case class Word (word: String, frequency: Int) val lines: DataStream[String] = env.fromSocketStream(...) lines.flatMap {line => line.split(" ") .map(word => Word(word,1))} .window(Time.of(5,SECONDS)).every(Time.of(1,SECONDS)) .groupBy("word").sum("frequency") .print() val lines: DataSet[String] = env.readTextFile(...) lines.flatMap {line => line.split(" ") .map(word => Word(word,1))} .groupBy("word").sum("frequency") .print() DataSet API (batch): DataStream API (streaming):
- 17. Technology Stacks You Know
- 18. But You Probably Didn’t Know About….
- 19. Apache Samza • Created by LinkedIn to address extend the capabilities of Apache Kafka • Simple API • Managed state • Fault Tolerant • Durable messaging • Scalable • Extensible • Processor Isolation
- 20. Apache Samza: Overview • Samza code runs as a Yarn job • You implement the StreamTask interface, which defines a process() call. • StreamTask runs inside a task instance, which itself is inside a Yarn container.
- 21. Apache Samza: Operators • Filter records matching condition • Map record ⇒ func(record) • Join two/more datasets by key • Group records with the same value in field • Aggregate records within the same group • Pipe job 1’s output ⇒ job 2’s input • MapReduce assumes fixed dataset. Can we adapt this to unbounded streams?
- 22. Apache Samza: Sample Code
- 24. Technology Stacks You Know
- 25. But You Probably Didn’t Know About….
- 26. Google Cloud Data Flow • Native Google Cloud data processing service • Simple programming model for batch and streamed data processing tasks • Provides a data flow managed service to control the execution of data processing jobs • Data processing jobs can be authored using the Data Flow SDKs (Apache Beam)
- 27. Google Cloud Data Flow : Details • A pipeline encapsulates an entire series of computations that accepts some input data from external sources, transforms that data produces some output data. • A PCollection abstracts a data unit in a pipeline • Sources and Sink abstract read and write operations in a pipeline • Google Data Flow provides management, monitoring and security capabilities in data pipelines
- 28. Google Cloud Data Flow is Based on Apache Beam • 1. Portable - You can use the same code with different runners (abstraction) and backends on premise, in the cloud, or locally • 2. Unified - Same unified model for batch and stream processing • 3. Advanced features - Event windowing, triggering, watermarking, lateless, etc. • 4. Extensible model and SDK - Extensible API; can define custom sources to read and write in parallel
- 29. But You Probably Didn’t Know About….
- 30. StreamSets Data Collector • Data processing platform optimized for data in motion • Visual data flow authoring model • Open source distribution model • On-premise and cloud distributions • Rich monitoring and management interfaces
- 31. StreamSets Data Collector: Details • Data collectors streams and process data in real time using data pipelines • A pipeline describes a data flow from origin to destination • A pipeline is composed of origins, destinations and processors • Extensibility model based on JavaScript and Jython • The lifecycle of a data collector can be controlled via the administration console
- 32. Machine Learning
- 33. Technology Stacks You Know
- 34. But You Probably Didn’t Know About….
- 35. TensorFlow • Second generation Machine Learning system, followed by DistBelief • TensorFlow grew out of a project at Google, called Google Brain, aimed at applying various kinds of neural network machine learning to products and services across the company. • An open source software library for numerical computation using data flow graphs • Used in following projects at Google 1. DeepDream 2. RankBrain 3. Smart Reply And many more..
- 36. TensorFlow: Details • Data flow graphs describe mathematical computation with a directed graph of nodes & edges • Nodes in the graph represent mathematical operations. • Edges represent the multidimensional data arrays (tensors) communicated between them. • Edges describe the input/output relationships between nodes. • The flow of tensors through the graph is where TensorFlow gets its name.
- 37. TensorFlow • Tensor • Variable • Operation • Session • Placeholder • TensorBoard
- 38. Fast Data Access
- 39. Technology Stacks You Know
- 40. But You Probably Didn’t Know About….
- 41. Druid • Druid was started in 2011 • ‣ Power interactive data applications • ‣ Multi-tenancy: lots of concurrent users • ‣ Scalability: trillions events/day, sub-second queries • ‣ Real-time analysis • Key Features • LOW LATENCY INGESTION • FAST AGGREGATIONS • ARBITRARY SLICE-N-DICE CAPABILITIES • HIGHLY AVAILABLE • APPROXIMATE & EXACT CALCULATIONS
- 42. Druid: Details • Realtime Node • Historical Node • Broker Node • Coordinator Node • Indexing Service
- 43. Druid: Details • Realtime Node • Historical Node • Broker Node • Coordinator Node • Indexing Service • JSON based query language
- 44. Low Latency Data Flows
- 45. Technology Stacks You Know
- 46. But You Probably Didn’t Know About….
- 47. Apache NiFi • Powerful and reliable system to process and distribute data • Directed graphs of data routing and transformation • Web-based User Interface for creating, monitoring, & controlling data flows • Highly configurable - modify data flow at runtime, dynamically prioritize data • Data Provenance tracks data through entire system • Easily extensible through development of custom components
- 49. Apache NiFi: Concepts • FlowFile • Unit of data moving through the system • Content + Attributes (key/value pairs) • Processor • Performs the work, can access FlowFiles • Connection • Links between processors • Queues that can be dynamically prioritized • Process Group • Set of processors and their connections • Receive data via input ports, send data via output ports
- 50. Data Discovery
- 51. Technology Stacks You Know
- 52. But You Probably Didn’t Know About…. WhereHows
- 53. Linkedin WhereHows • Where is my data? How did it get there? • Enterprise data catalog • Metadata search • Collaboration • Data lineage analysis • Connectivity to many data sources and ETL tools • Powering Linkedin data discovery layer
- 54. Linkedin WhereHows: Architecture • Web interface for data discovery • API enabled • Backend server that controls the metadata crawling and integration with other systems
- 55. Linkedin WhereHows: Data Lineage • Collects metadata from ETL platforms and scripts • Sources include • Pig • MapReduce • Informatica • Teradata • Visualizes the lineage information associated with a data source
- 57. Technology Stacks You Know
- 58. But You Probably Didn’t Know About….
- 59. Microsoft Cognitive Services • Based on Project Oxford and Bing • Offers 22 cognitive computing APIs • Main categories include: • Vision • Speech • Language • Knowledge • Search • Integrated with Cortana Intelligence Suite
- 61. Microsoft Cognitive Services: Developer Experience • 22 different REST APIs that abstract cognitive capabilities • SDKs for Windows, IOS, Android and Python • Open source
- 62. Summary • The big data ecosystem is constantly evolving • There are a lot of relevant new technologies beyond the traditional Hadoop-Spark stacks • Big internet companies are leading innovation in the space