Talend introduction v1

Make (big) data meaningfulSoftnix Technology
Softnix
Big Data
Platform
Company
Talend
Introduction

Agenda
• Introduction to Softnix
• Data Engineer vs Data Scientist
• Data Lake
• Talend Big Data Integration
• Demo

Logger Logger Cloud
for MSP
Data PlatformAuthenticator
Logger for AWS
Logger for Azure
“Big Data Platform Company”
Collector
Edge Point
All-in-one Law & Standard Compliance Security & IT Services
Monitoring by ZABBIX
Big Data Analytics
Products Module
- SDP for Log Analytic
- SDP for Cybersecurity
- SDP for IT Operation
& Monitoring
Product Module
- ISO 27001 Report

Technology Partner
Leader
Big Data Operation Technology
Leader
Data Integration
Data Quality
ISV Partner

Softnix Data Platform
Big Data Analytic Platform
Any Device
Any Platform
Dashboard & VisualizeEnterprise Data Lake
What ‘s Softnix Data Platform

Softnix Data Platform - Services
Framework

Architecture Softnix Data Platform
Architecture

Dashboard System

Source : https://www.datacamp.com/community/blog/data-scientist-vs-data-engineer

Roles Big Data Project
Roles Responsibility Example Jobs Type of analytic Weakness
Data Analyst Data Query, Process,
Summary, Visualize
Data
Create Monthly Report,
Summary Report by
business request
Backward (Descriptive,
Diagnostic )
Not have skill to analytic
and management Big
Data
Data Engineer Create pipeline data for
serve to business
requirement, Clean,
Accurate date, Prepare
data to data analyst or
data scientist,
Knowledge of data
warehouse, data lake,
data modeling
Setup server,
maintenance big data
server, collect data,
Data Preparation,
Import and export data
to easy analyst format
Backward and forward Not have analyst data
skill
Data Scientist Work with machine
learning, AI to analytic
data
Work with raw data,
focus to insight of data,
find new meaningful
Focus to Forward
(Predictive &
Prescriptive)
Operation Big data
infrastructure

What data scientists spend the most time doing
Data preparation accounts for about 80% of
the work of data scientists
Source: https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-
survey-says/#2bdb12bf6f63

Why Do We Need a Data Lake?
“Data lakes are enterprise-wide data management platforms for analyzing
disparate sources of data in its native format.”, Gartner.
BusinessValue
Reducing cost
Generating new opportunities
• ETL offload
• EDW offload/optimization
• Data archiving
• Customer acquisition,
retention..
• Real-time engagement
• Pricing optimization
• Demand forecasting
• Risk and fraud
• Predictive maintenance
• Smart products…

The rest
of us
Data Lakes Bring New Challenges
High-end
users
Complexity, poor governance and control, no reuse

Data Lake – Conceptual
Architecture
Acquire
Ingest
Understand
& Improve
Curate &
Govern
Deliver
Self-
service
SCALE

Ingestion Best Practices
Transactions
Messages & Events
1011
0111
0010
1011
0111
0010
Logs
Sensors
Data Analytics & Data Science
Real-time Data Visualization
Real-time Indicators / Scorecard
Collect - Distribute
Track
Streaming
Alert
NYC Taxi Data Streaming

• Hadoop: the core project
• HDFS: the Hadoop Distributed File System
• MapReduce: the software framework for distributed processing of
large data sets
• Hive: a data warehouse infrastructure that provides data
summarization and a querying language
• Pig: a high-level data-flow language and execution framework for
parallel computation
• HBase: this is the Hadoop database. Use it when you need
random, realtime read/write access to your Big Data
• And many many more: Sqoop, HCatalog, Zookeeper, Oozie,
Cassandra, MongoDB, Flume, Impala, Stinger, Neo4J, etc.
Big Data Ecosystem

Talend Big Data Integration

GARTNER DATA INTEGRATION MAGIC QUADRANT 2017

Talend Unique Integration
Solution

• Basic Concept
• Monitor the hadoop cluster
• Create cluster metadata
• Reading and writing data in HDFS
• reading and writing to HDFS and Hbase
• Working with tables
• Import DB table with Scoop
• Create table with Hive
• Processing data and table in HDFS
• Processing tables with Hive
• Processing data with Pig
• Processing data with Big Data batch jobs (MapReduce or Spark)
Talend Big Data Capabilities

Talend Product Strategy

Data Preparation
Components
• Select – include/exclude data
• Merge – combine data from multiple source
• Clean – handle missing value, duplicates, null value
• Transform – convert data into analysis format
• Enrich – derive attributes, generate fields
• Aggregate – convert transactional rows of data into
summarize columns.

Vision: Democratize big data

Talend Big Data

Typical Use Case Scenario

Hadoop Connector

• Step 1 Import File into Cloudera Hadoop
• Step 2 Preparation Process on HDFS
• Step 3 Setup Hive Table
• Step 4 Load file to Hive
• Step 5 Using Hue to explore data
Demo

Talend introduction v1

More Related Content

What's hot (20)

Similar to Talend introduction v1 (20)

Recently uploaded (20)

Talend introduction v1