Next-Generation BPM - How to create intelligent Business Processes thanks to Big Data and Apache Hadoop

Next-Generation BPM –
How to create intelligent Business Processes
thanks to Big Data
Talend, Global Leader in Open Source Integration Solutions

Kai Wähner
kontakt@kai-waehner.de
@KaiWaehner
Xing / LinkedIn
www.kai-waehner.de

Kai Wähner
Main Tasks
Requirements Engineering
Enterprise Architecture Management
Business Process Management

Architecture and Development of Applications
Service-oriented Architecture
Integration of Legacy Applications
Cloud Computing

Big Data

Contact
Consulting
Developing
Coaching
Speaking
Writing
© Talend 2013

Email: kontakt@kai-waehner.de

Blog: www.kai-waehner.de/blog
Twitter: @KaiWaehner
Social Networks: Xing, LinkedIn

“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Key messages

BPM should be used (just) for optimizing business processes!
Intelligent business processes need big data and integration!
Big data will reduce human interactions in BPM further!
© Talend 2013


Agenda
• Big data paradigm shift
• Use cases for big data
• Intelligent business processes
• Technology and product perspective
• Implementation of an use case

© Talend 2013


Why should you care about big data?

“If you can't measure it,
you can't manage it.”
William Edwards Deming
(1900 –1993)
American statistician, professor,
author, lecturer and consultant

© Talend 2013


Why should you care about big data?

„Silence the HiPPOs“ (highest-paid person‘s opinion)
 Being able to interpret unimaginable large data
stream, the gut feeling is no longer justified!


© Talend 2013


Why does big data exist?
Changing Scale

Sensors

Changing Expectations

Cloud

Changing Interactions
© Talend 2013


Three shifts in the way we analyze information
• Messiness: Using ALL data, not just samples
• Also bad data (e.g. Word spell checker, Google autocomplete and „did you mean...“ recommendation

• Correlations: Instead of causalities
• May not tell us WHY something is happening, but THAT it
is happening
• In many situations, this is good enough
• What drug substance cures cancer? When should I buy an
airplane ticket?

• Datafication: Store, process, combine, reuse,
enhance all data!
• Digitalisation (Amazon Kindle  Read) vs. Datafication
(Google Books  Read, Search, Process, ...)
• Words becomes data: Google books: not just read, but
also search, analyse, etc.
• Locations becomes data: GPS: not just navigation, but
also insurance costs, economic routes, etc.

© Talend 2013


The Vs of big data

Volume
(terabytes,
petabytes)

Velocity
(realtime or nearrealtime)

Variety
(social networks,
blog posts, logs,
sensors, etc.)

© Talend 2013


Value

Big data tasks to solve - before analysis
Big Data Integration
– Land data in a Big Data cluster
– Implement or generate parallel processes

Big Data Manipulation
– Simplify manipulation, such as sort and filter
– Computational expensive functions

Big Data Quality & Governance
– Identify linkages and duplicates, validate big data
– Match component, execute basic quality features

Big Data Project Management
– Place frameworks around big data projects
– Common Repository, scheduling, monitoring
© Talend 2013


Storage  Reduce costs

Global Parcel Service
A lot of data must be stored „forever“
➜ Numbers increase exponentially
➜ Goal: As cheap as possible
➜ Problem: (Fast) queries must still be possible
➜ Solution: Commodity servers and „Hadoop querying“
➜

http://archive.org/stream/BigDataImPraxiseinsatz-SzenarienBeispieleEffekte/Big_Data_BITKOM-Leitfaden_Sept.2012#page/n0/mode/2up

© Talend 2013


Replace ETL Improve performance

“The advantage of their new system is that they can now look at their data
[from their log processing system] in anyway they want:
➜ Nightly MapReduce jobs collect statistics about their mail system such as
spam counts by domain, bytes transferred and number of logins.
➜ When they wanted to find out which part of the world their customers
logged in from, a quick [ad hoc] MapReduce job was created and they had
the answer within a few hours. Not really possible in your typical ETL
system.”
http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data
© Talend 2013


Risk management  Customer success

Deduce
Customer
Defections

http://hkotadia.com/archives/5021

© Talend 2013


Flexible pricing  Increase revenue

➜

➜
➜

➜

With revenue of almost USD 30 billion and a network of
800 locations, Macy's is considered the largest store operator in the
USA
Daily price check analysis of its 10,000 articles in less than two hours
Whenever a neighboring competitor anywhere between New York
and Los Angeles goes for aggressive price reductions, Macy's follows
its example
If there is no market competitor, the prices remain unchanged

http://www.t-systems.com/about-t-systems/examples-of-successes-companies-analyze-big-data-in-record-time-l-t-systems/1029702

© Talend 2013


Great big data use cases, but ...
➜

➜
➜

➜

How do you put this big data easily in the hands of the people that
need it?
Making the data “actionable” is the real challenge.
Seeing the information that helps make a decision on a composite
dashboard is just the first step and where too many companies stop.
A business must be able to fire off the business process to execute the
decision made regarding the data.

http://smartdatacollective.com/matt-davies/104576/data-driven-bpm-making-big-data-actionable
© Talend 2013


Kai Wähner

Intelligent business processes
Humans have to interpret large data to make decision.
Using gut feeling is nothing but gambling.
➜ Just doing big data analytics is not enough. Systematic
and monitored human interactions are as important to
get best outcomes.
➜ An intelligent business process
combines big data and BPM. This
enables humans to make data-driven
decisions based on big data analytics.
➜

© Talend 2013


Intelligent business processes
➜

Process starts action (PULL Big Data)
• Manual or automated
• Faster responses (e.g. „spam by domain“)
• Better outcomes (e.g. „recommendation engine“)

➜

Data starts action (Big Data PUSH)
• (Usually) automated
• Predictive processes (e.g. „preventing flu epidemic“)
• Handle before it happens (e.g. „customer deduction“)

© Talend 2013


Combination of big data and BPM
How are they related?
➜ How to combine?
➜ How to realize this technically?
➜

© Talend 2013


How BPM?
Script Task

Groovy
JavaScript
etc.

Service Task

SOAP Web Service

Everything
from Cobol
to Ruby...
© Talend 2013


Kai Wähner

... or a
„big data
service“

Challenge

Separation of Concerns
© Talend 2013


Kai Wähner

Building Blocks for „Intelligent Business Processes“

Integration
• ETL
• Connectivity / adaptors to connect to external
systems using a variety of different protocols
• Predefined EIP for message routing

Big Data
• Processing
• Analytics

BPM
• Do queries to make decisions
• Human or machine

© Talend 2013


Kai Wähner


Integration
• Extract Transform Load (ETL)

Big Data
• Processing
• Analytics

BPM

© Talend 2013


Kai Wähner

Enterprise Integration Patterns

© Talend 2013


Do not write all that “glue code”!
AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(
S3Sample.class.getResourceAsStream("AwsCredentials.properties")));
String bucketName = "my-first-s3-bucket-" + UUID.randomUUID();
String key = "MyObjectKey";
try {

s3.createBucket(bucketName);
s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile()));
S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
.withBucketName(bucketName)
.withPrefix("My"));
s3.deleteObject(bucketName, key);
s3.deleteBucket(bucketName);
} catch (AmazonServiceException ase) {
// error handling...
} catch (AmazonClientException ace) {
// error handling...
}
© Talend 2013


Integration framework (e.g. Apache Camel)
// Producer
from(“ftp:toS3")
.setHeader(S3Constants.KEY, simple(“order.txt"))
.to("aws-s3://myBucket?accessKey=" + a+ "&secretKey= " + s)

// Consumer
from(„salesforce://orders__c?user=dummy1“)
.filter(„customer == ${dummyCustomer})
.to(“ibm-database:orderData")

© Talend 2013


Enterprise Service Bus (e.g. Talend ESB)

© Talend 2013


Alternatives for integration

Integration Suite

Enterprise
Service Bus

Integration
Framework

Low

Connectivity
Routing
Transformation

© Talend 2013

High

+

INTEGRATION
Tooling
Monitoring
Support

+

BUSINESS PROCESS MGT.
BIG DATA / MDM
REGISTRY / REPOSITORY
RULES ENGINE
„YOU NAME IT“


Complexity
of Integration

Technology perspective

How to process big data?
© Talend 2013



The defacto standard for big data processing

© Talend 2013



“A big part of [the
company’s strategy]
includes wiring SQL Server
2012 (formerly known by
the codename “Denali”) to
the Hadoop distributed
computing platform, and
bringing Hadoop to
Windows Server and Azure”

Even Microsoft (the .NET house) relies on Hadoop since 2011
© Talend 2013


What is Hadoop?

Apache Hadoop, an open-source software library, is a
framework that allows for the distributed processing of
large data sets across clusters of commodity hardware
using simple programming models. It is designed to scale
up from single servers to thousands of machines, each
offering local computation and storage.
© Talend 2013


Hadoop alternatives

Integration Suite

Hadoop
Distribution

Apache
Hadoop

few

MapReduce
HDFS
Ecosystem

© Talend 2013

many

+

Packaging
Deployment-Tooling
Support

Tooling / Modeling
Code Generation
Scheduling
Other Tools (ESB, BPM, ...)

+


Features
included

(Near) Realtime?

!
Hadoop cannot solve every big data problem.
Complex event processing and real-time analytics have
to be solved in another way (at least today).
In-memory computing and streaming platforms are good
alternatives or complements to Hadoop for processing
and analyzing big data.
© Talend 2013


BPMN
„Business Process Model and Notation (BPMN) is a graphical
representation for specifying business processes in a business process
model.“  BPMN 2.0 is also executable!
Wikipedia

© Talend 2013


Kai Wähner

Alternatives for BPM

Integration Suite

BPM Suite

BPM
Framework

Low

High

Coding
Service Tasks
Human Interaction
GUI

© Talend 2013

+

BPM
Tooling
Monitoring
Support

+

ESB
BIG DATA / MDM
REGISTRY / REPOSITORY
RULES ENGINE
„YOU NAME IT“


Kai Wähner

Complexity of
Orchestration

Frameworks vs. Tools

Suite of Tools
Specific Tools
Frameworks

Low

High

e.g.
Camel (Integration)
Hadoop (Big Data)
Activiti (BPM)
© Talend 2013

e.g.
Mule ESB (Integration)
MapR (Big Data)
Camunda (BPM)

e.g.
Talend Unified Platform
i.e ALL-IN-ONE
(Integration, Big Data, BPM)


Kai Wähner

Complexity of
Orchestration

Custom combination of integration, big data and BPM?

• A lot of glue code
• Testing
• Bugfixing
• No support
Some other people already had
the problems you would have!

© Talend 2013


Kai Wähner

Implementation of an use case

Suite of Tools
Specific Tools
Frameworks

Low

High

e.g.
Camel (Integration)
Hadoop (Big Data)
Activiti (BPM)
© Talend 2013

e.g.
Mule ESB (Integration)
MapR (Big Data)
Camunda (BPM)

Complexity of
Orchestration

e.g.
i.e ALL-IN-ONE
(Integration, Big Data, BPM)



Big Data

Data
Quality

Data
Integration

MDM

ESB

BPM

 Commercial license
 Subscription model
 Support included
 Open source license
 Free of charge
Big
Data

Data
Quality

Data
Integration

MDM

ESB

 Optional support
 Based on open source

projects such as Eclipse
or Apache Camel, CXF,
Hadoop

© Talend 2013


Example (Talend): Integration

Connect to data sources from competitors, for example via REST service,
Twitter API, or custom scripts.
© Talend 2013


Example (Talend): Big Data Processing

Move data to HDFS for processing, as your classic servers and data warehouses are
not able to process this semi-structured data fast enough (and cheap), probably.
Manipulate the data, in other words, filter relevant information, sort it, and
compare it to prices of your products.
© Talend 2013


Example (Talend): Business Process

Start a new instance of a business process to review the result and continue
with further tasks, such as calling a web service which does the price reduction
in selected locations. Reviews can be done by human interaction or via
automated tasks depending on the proposed price reduction.
© Talend 2013


Thank you for your attention. Questions?
KAI WÄHNER
kontakt@kai-waehner.de
www.kai-waehner.de
LinkedIn / Xing
@KaiWaehner

Next-Generation BPM - How to create intelligent Business Processes thanks to Big Data and Apache Hadoop

More Related Content

What's hot (18)

Viewers also liked (20)

Similar to Next-Generation BPM - How to create intelligent Business Processes thanks to Big Data and Apache Hadoop (20)

More from Kai Wähner (20)

Recently uploaded (20)

Next-Generation BPM - How to create intelligent Business Processes thanks to Big Data and Apache Hadoop