How to create intelligent Business Processes thanks to Big Data (BPM, Apache Hadoop, Talend, Open Source)
23 likes9,260 views
The document discusses how to create intelligent business processes using big data, emphasizing the importance of BPM for optimizing these processes. Key insights include the need for big data integration, manipulation, quality management, and governance, along with the role of humans in interpreting data for decision-making. Additionally, it highlights the combination of big data and BPM to enhance efficiency, quality, and flexibility while reducing costs.
![©
Talend
2013
“How
to
create
intelligent
Business
Processes
thanks
to
Big
Data”
by
Kai
Wähner
Why should you care about big data?
“Accessing
data
is
now
[again]
the
cri7cal
path
in
making
good
decisions!”
h`p://www.inforbix.com/friday-‐data-‐stories-‐big-‐data-‐driven-‐decision-‐making/](https://image.slidesharecdn.com/201306bpmconintelligentbpswithbigdatakaiwaehner-130613022448-phpapp01/85/How-to-create-intelligent-Business-Processes-thanks-to-Big-Data-BPM-Apache-Hadoop-Talend-Open-Source-8-320.jpg)
![©
Talend
2013
“How
to
create
intelligent
Business
Processes
thanks
to
Big
Data”
by
Kai
Wähner
Even
MicrosoL
(the
.NET
house)
relies
on
Hadoop
since
2011
How to process big data?
“A
big
part
of
[the
company’s
strategy]
includes
wiring
SQL
Server
2012
(formerly
known
by
the
codename
“Denali”)
to
the
Hadoop
distributed
compu7ng
plasorm,
and
bringing
Hadoop
to
Windows
Server
and
Azure”](https://image.slidesharecdn.com/201306bpmconintelligentbpswithbigdatakaiwaehner-130613022448-phpapp01/85/How-to-create-intelligent-Business-Processes-thanks-to-Big-Data-BPM-Apache-Hadoop-Talend-Open-Source-40-320.jpg)
How to create intelligent Business Processes thanks to Big Data (BPM, Apache Hadoop, Talend, Open Source)
- 1. Next-Generation BPM – How to create intelligent Business Processes thanks to Big Data Talend, Global Leader in Open Source Integra7on Solu7ons Kai Wähner kwaehner@talend.com @KaiWaehner Xing / LinkedIn www.kai-waehner.de
- 2. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Consulting Developing Coaching Speaking Writing Main Tasks Requirements Engineering Enterprise Architecture Management Business Process Management Architecture and Development of Applications Service-oriented Architecture Integration of Legacy Applications Cloud Computing Big Data Contact Email: kontakt@kai-waehner.de Blog: www.kai-waehner.de/blog Twitter: @KaiWaehner Social Networks: Xing, LinkedIn Kai Wähner
- 3. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Key messages BPM should be used (just) for optimizing business processes! Intelligent business processes need big data and integration! Big data will reduce human interactions in BPM further!
- 4. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda
- 5. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda
- 6. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner William Edwards Deming (1900 –1993) American sta7s7cian, professor, author, lecturer and consultant “If you can't measure it, you can't manage it.” Why should you care about big data?
- 7. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner è „Silence the HiPPOs“ (highest-‐paid person‘s opinion) è Being able to interpret unimaginable large data stream, the gut feeling is no longer jus7fied! Why should you care about big data?
- 8. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Why should you care about big data? “Accessing data is now [again] the cri7cal path in making good decisions!” h`p://www.inforbix.com/friday-‐data-‐stories-‐big-‐data-‐driven-‐decision-‐making/
- 9. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Why does big data exist? Changing Interactions Changing Scale Changing Expectations Cloud Sensors
- 10. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner The Vs of big data Volume (terabytes, petabytes) Variety (social networks, blog posts, logs, sensors, etc.) Velocity (real7me or near-‐ real7me) Value
- 11. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Big Data Integra3on – Land data in a Big Data cluster – Implement or generate parallel processes Big Data Manipula3on – Simplify manipula7on, such as sort and filter – Computa7onal expensive func7ons Big Data Quality & Governance – Iden7fy linkages and duplicates, validate big data – Match component, execute basic quality features Big Data Project Management – Place frameworks around big data projects – Common Repository, scheduling, monitoring Big data tasks to solve - before analysis
- 12. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda
- 13. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner h`p://www.slideshare.net/brocknoland/common-‐and-‐unique-‐use-‐cases-‐for-‐apache-‐hadoop Replacing ETL jobs: Binary files
- 14. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner h`p://hkotadia.com/archives/5021 Deduce Customer Defec7ons Forecast: Risk management
- 15. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ With revenue of almost USD 30 billion and a network of 800 loca7ons, Macy's is considered the largest store operator in the USA ➜ Daily price check analysis of its 10,000 ar7cles in less than two hours ➜ Whenever a neighboring compe7tor anywhere between New York and Los Angeles goes for aggressive price reduc7ons, Macy's follows its example ➜ If there is no market compe7tor, the prices remain unchanged h`p://www.t-‐systems.com/about-‐t-‐systems/examples-‐of-‐successes-‐companies-‐analyze-‐big-‐data-‐in-‐record-‐7me-‐l-‐t-‐systems/1029702 Logistics: Flexible pricing
- 16. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Content manipula7on at a traveling portal – Which customer reviews are trustworthy? Jean-‐Paul Schmetz, Harvard Business Manager, November 2012 Fraud detection: Fraud mining
- 17. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ A lot of data must be stored „forever“ ➜ Numbers increase exponen7ally ➜ Goal: As cheap as possible ➜ Problem: (Fast) queries must s7ll be possible ➜ Solu7on: Commodity servers and „Hadoop querying“ Global Parcel Service h`p://archive.org/stream/BigDataImPraxiseinsatz-‐SzenarienBeispieleEffekte/Big_Data_BITKOM-‐Leisaden_Sept.2012#page/n0/mode/2up Storage: Compliance
- 18. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ How do you put this big data easily in the hands of the people that need it? ➜ Making the data “ac7onable” is the real challenge. ➜ Seeing the informa7on that helps make a decision on a composite dashboard is just the first step and where too many companies stop. ➜ A business must be able to fire off the business process to execute the decision made regarding the data. Great big data use cases, but ... Kai Wähner h`p://smartdatacollec7ve.com/ma`-‐davies/104576/data-‐driven-‐bpm-‐making-‐big-‐data-‐ac7onable
- 19. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda
- 20. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ Humans have to interpret large data to make decision. Using gut feeling is nothing but gambling. ➜ Just doing big data analy7cs is not enough. Systema7c and monitored human interac7ons are as important to get best outcomes. ➜ An intelligent business process combines big data and BPM. This enables humans to make data-‐driven decisions based on big data analy7cs. Intelligent business processes
- 21. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ Process starts ac7on (Big Data PULL) • Manual or automated • Faster responses (e.g. „catching Osama“) • Be`er outcomes (e.g. „recommenda7on engine“) ➜ Data starts ac7on (Big Data PUSH) • (Usually) automated • Predic7ve processes (e.g. „preven7ng flu epidemic“) • Handle before it happens (e.g. „customer deduc7on“) Intelligent business processes
- 22. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ How are they related? ➜ How to combine? ➜ How to realize this technically? Combination of big data and BPM
- 23. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner increase efficiency better quality reduce costs increase flexibility enable new business models Why BPM? Kai Wähner Business-IT-Alignment
- 24. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • long-running stateful workflows • frequently changing processes • human interaction When BPM? Kai Wähner
- 25. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner How BPM? Kai Wähner Script Task Service Task Groovy JavaScript etc. SOAP Web Service Everything from Cobol to Ruby... ... or a „big data service“
- 26. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Challenge Kai Wähner Separation of Concerns
- 27. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Separation of Concerns Kai Wähner Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine C R E A T E U S E
- 28. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda
- 29. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine Technical / product perspective
- 30. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Enterprise Integration Patterns
- 31. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Enterprise Integration Patterns
- 32. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner AmazonS3 s3 = new AmazonS3Client(new Proper7esCreden7als( S3Sample.class.getResourceAsStream("AwsCreden7als.proper7es"))); String bucketName = "my-‐first-‐s3-‐bucket-‐" + UUID.randomUUID(); String key = "MyObjectKey"; try { s3.createBucket(bucketName); s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile())); S3Object object = s3.getObject(new GetObjectRequest(bucketName, key)); ObjectLis7ng objectLis7ng = s3.listObjects(new ListObjectsRequest() .withBucketName(bucketName) .withPrefix("My")); s3.deleteObject(bucketName, key); s3.deleteBucket(bucketName); } catch (AmazonServiceExcep7on ase) { // error handling... } catch (AmazonClientExcep7on ace) { // error handling... } Integration framework
- 33. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner // Producer from(“ftp:toS3") .setHeader(S3Constants.KEY, simple(“order.txt")) .to("aws-s3://myBucket?accessKey=" + a+ "&secretKey= " + s) // Consumer from(„salesforce://orders__c?user=dummy1“) .filter(„customer == ${dummyCustomer}) .to(“ibm-database:orderData") Integration framework (e.g. Apache Camel)
- 34. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Enterprise Service Bus (e.g. Talend ESB)
- 35. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Connectivity Routing Transformation Complexity of Integration Enterprise Service Bus Integra7on Suite Low High Integration Framework INTEGRATION Tooling Monitoring Support+ BUSINESS PROCESS MGT. BIG DATA / MDM REGISTRY / REPOSITORY RULES ENGINE „YOU NAME IT“ Alternatives for integration +
- 36. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine Technical / product perspective
- 37. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Technology perspective How to process big data?
- 38. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Slides: h`p://www.slideshare.net/pavlobaron/100-‐big-‐data-‐0-‐hadoop-‐0-‐java Video: h`p://www.infoq.com/presenta7ons/Big-‐Data-‐Hadoop-‐Java How to process big data?
- 39. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner The defacto standard for big data processing How to process big data?
- 40. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Even MicrosoL (the .NET house) relies on Hadoop since 2011 How to process big data? “A big part of [the company’s strategy] includes wiring SQL Server 2012 (formerly known by the codename “Denali”) to the Hadoop distributed compu7ng plasorm, and bringing Hadoop to Windows Server and Azure”
- 41. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Apache Hadoop, an open-‐source soLware library, is a framework that allows for the distributed processing of large data sets across clusters of commodity hardware using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computa7on and storage. What is Hadoop?
- 42. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner How to process big data?
- 43. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner MapReduce HDFS Ecosystem Features included Hadoop Distribu7on Integra7on Suite few many Apache Hadoop Packaging Deployment-Tooling Support + Tooling / Modeling Code Generation Scheduling Other Tools (ESB, BPM, ...) + Hadoop alternatives
- 44. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine Technical / product perspective
- 45. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Standards Kai Wähner BPM BPEL BPMN XPDL WF-XML BPEL4People jPDL ARIS EPC
- 46. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner BPMN Kai Wähner „Business Process Model and Notation (BPMN) is a graphical representation for specifying business processes in a business process model.“ Wikipedia
- 47. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • BPMN is a standard notation for designing business processes (versus: UML is a standard modeling language best suited for designing and implementing software) • not just flow charts! sufficient restrictions / constraints => executable! • standardized XML format • orchestration and choreography • extension points => add specific needs without breaking interoperability • optional mapping of a BPMN subset to BPEL (restricted to block-structured flows without cycles) BPMN 2.0 Kai Wähner
- 48. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Alternatives for BPM (technology perspective) Kai Wähner Complexity of Orchestration Low High BPM Framework BPM Suite Integra7on Suite Coding Service Tasks Human Interaction GUI BPM Tooling Monitoring Support + ESB BIG DATA / MDM REGISTRY / REPOSITORY RULES ENGINE „YOU NAME IT“ +
- 49. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine Let‘s realize it !!! Technical / product perspective
- 50. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Alternatives for creating intelligent BPs Kai Wähner Complexity of Orchestration Low High Frameworks Specific Tools Suite of Tools e.g. Talend Unified Platform i.e ALL-IN-ONE (Integration, Big Data, BPM) e.g. Camel (Integration) Hadoop (Big Data) Activiti (BPM) e.g. Mule ESB (Integration) MapR (Big Data) Camunda (BPM)
- 51. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Custom combination of integration, big data and BPM? Kai Wähner • A lot of glue code • Testing • Bugfixing • No support Some other people already had the problems you would have!
- 52. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda
- 53. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Live demo Complexity of Orchestration Low High Frameworks Specific Tools Suite of Tools e.g. Talend Unified Platform i.e ALL-IN-ONE (Integration, Big Data, BPM) e.g. Camel (Integration) Hadoop (Big Data) Activiti (BPM) e.g. Mule ESB (Integration) MapR (Big Data) Camunda (BPM)
- 54. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Data Quality Data Integration MDM ESB Big Data Big Data ¾ Open source license ¾ Free of charge ¾ OpDonal support ¾ Based on open source projects such as Eclipse or Apache Camel, CXF, Hadoop ¾ Commercial license ¾ SubscripDon model ¾ Support included Data Quality Data Integration MDM ESB BPM Talend Unified Platform
- 55. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner „Talend Integra7on Suite“ in ac7on... Live demo
- 56. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Example (Talend): Integration
- 57. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Example (Talend): Big Data Processing
- 58. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Example (Talend): Business Process
- 59. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Example (Talend): Big Data Web Service
- 60. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Did you get the key message?
- 61. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Key messages BPM should be used (just) for optimizing business processes! Intelligent business processes need big data and integration! Big data will reduce human interactions in BPM further!
- 62. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Did you get the key message?
- 63. Thank you for your attention. Questions? KAI WÄHNER kwaehner@talend.com www.kai-waehner.de LinkedIn / Xing @KaiWaehner