HPCC Systems Engineering Summit: Community Use Case: Because Who Has Time for MapReduce?
0 likes603 views
The document discusses the use of HPCC systems and its ECL language for big data processing, highlighting its efficiency compared to traditional BI tools and Hadoop. It details projects at Citrix that utilize HPCC for ETL jobs, demonstrating that it outperforms Oracle in speed and capability. The author emphasizes HPCC's productivity and effectiveness in extracting insights from various data types.
HPCC Systems Engineering Summit: Community Use Case: Because Who Has Time for MapReduce?
- 1. Using HPCC Systems for Big Data and More -‐ Because Who Has Time for MapReduce? John Andleman October 7, 2014
- 2. About Me I love to architect and build systems that acquire, manage, and use data to solve problems • OperaKonal • AnalyKcal • Real-‐Kme • Big Data • Data Science
- 3. About Citrix SaaS Division a market-‐leading global provider of web collabora<on, remote access, data sharing and IT support so>ware as a service. GoToMee<ng For online mee*ngs GoToWebinar For do-‐it-‐yourself webinars GoToTraining For online training GoToAssist for integrated IT support tools GoToMyPC for remote access to your Mac or PC Sharefile for data sharing and storage Podio for social collabora*on OpenVoice for affordable audio conferencing
- 4. Finding Insights In Big Data • Structured and semi-‐structured data • Data sets from very small to many billions of records • Hundreds of terabytes of log files • Thousands of Oracle database tables • Spreadsheet data • And more…
- 5. Finding Insights In Big Data – TradiKonal BI? • Oracle Data Warehouse • ROLAP and Data Cubes • Very expensive licensing and hardware costs • Does not scale well to very large data sets • ETL to get data loaded is complicated • ExtracKng useful content from log files is complicated • Limited analyKc capabiliKes
- 6. Finding Insights In Big Data – Hadoop? • It’s very powerful, but… • Why do they have to make it so complicated?! • MapReduce scales, but it is a giant step backwards in producKvity • Java is a horrible language for data processing; Python is a li[le be[er • ExtracKng useful content from log files is very complicated • Much of the Hadoop infrastructure is immature and poorly documented
- 7. Finding Insights In Big Data – Hadoop with Pig? • Much more producKve than wriKng MapReduce code, but… • The language is very limited • Where the language has gaps, you end up wriKng user-‐defined funcKons, or worse, going back to wriKng MapReduce code • ExtracKng useful content from log files is sKll very complicated
- 8. Finding Insights In Big Data – HPCC? • ECL Language is a very mature data processing language • ECL is a very complete language • ECL has very powerful pa[ern matching constructs for extracKng useful content from log files – the best I have seen! • ECL is the best ETL language I have worked with • HPCC with ECL scales well and is a very producKve development environment
- 9. Big Data Projects at Citrix – GoToMeeKng and GoToWebinar • Study product feature usage by: ᵒ Different customer segments ᵒ Trial vs. paid customers ᵒ Retained vs. lost accounts • Study trial usage pa[erns of converted vs. non-‐converted accounts • Study relaKonships of various session staKsKcs to customer retenKon • Study usage pa[erns of VoIP vs. dial-‐up audio in sessions • Study pa[erns of session audio problems • Study to idenKfy fraudulent usage including trial abuse and spam acKvity
- 10. Big Data Projects at Citrix – HPCC and ECL • Raw Oracle data was dumped to CSV files for ingesKon into HPCC – this turned out to be faster than doing any data crunching in Oracle • HPCC for ETL jobs ran MUCH faster than Oracle, even when HPCC was run on much smaller hardware • HPCC is a much more capable and producKve ETL tool than anything else I have used. I even prefer it for data that is not “big”. • ECL has excellent support for analyKcs, especially on very large data sets that would choke most other analyKc tools
- 11. Work be[er. Live be[er.