Bringing Real-Time to the Enterprise with Hortonworks DataFlow
4 likes616 views
This document discusses TELUS's journey to enable real-time streaming analytics of data from IPTV set top boxes (STBs) to improve the customer experience. It describes moving from batch processing STB log data every 12 hours to streaming the data in real-time using Apache Kafka, NiFi, and Spark. Key lessons learned include using Java 8 for SSL, Spark 2.0 for Kafka integration, and addressing security challenges in their multi-tenant Hadoop environment.
Bringing Real-Time to the Enterprise with Hortonworks DataFlow
- 1. Bringing Real-Time to the Enterprise with Hortonworks DataFlow Cavan Loughran – TELUS Oliver Meyn – T4G June 2017
- 2. 2TELUS Public Introduction to TELUS and Optik TV® Starting state Target state Journey Lessons learned Agenda / Today’s Talk 1 2 3 4 5
- 3. TELUS Public 3 TELUS is Canada’s fastest-growing national telecommunications company, with $12.9B of annual revenue and 12.7M customer connections. TELUS provides Optik TV® to 1.1M customers At TELUS, our goal is to delight our customers by continuously improving the service/viewing experience. TELUS and Optik TV®
- 4. TELUS Public 4 Watching TV @Home Content Provider eg. NBC Service Creation Service Delivery Encoding & DRM IP/Internet/CDN Network Access Technology Eg. Fibre/Copper Your Neighborhood Your Home Your TV Home Gateway IPTV STB (Smart Device) Video flows as data over IP Streaming video monitoring
- 5. TELUS Public 5 IPTV Set Top Box (STB)
- 6. TELUS Public 6 STB’s report diagnostic logs every 12 hours. Daily batch load into Datalake using scp & pig. Through daily analysis we create and action key insights for the network and individual services. Starting State
- 7. TELUS Public 7 12 hour interval? TV is a streaming real-time service, a check point every 12 hours hides too many things from analysis and timely action.
- 8. TELUS Public 8 Identify and take action to correct a problem within minutes. Predict when a device and/or service is in danger of degrading or failing. How to achieve Target State: – Increase STB reporting frequency ~50X from twice a day to 96 per day. – Streaming analytics (ML) to monitor each STB and take immediate action. Target State
- 9. TELUS Public 9 Watching TV @Home Content Provider eg. NBC Service Creation Service Delivery Encoding & DRM IP/Internet/CDN Network Access Technology Eg. Fibre/Copper Your Neighborhood Your Home Your TV Home Gateway IPTV STB (Smart Device) Video flows as data over IP Streaming video monitoring Target state
- 11. TELUS Public 11 TELUS supports many enterprise use cases with a single “Datalake”, aka multi-tenancy. HDP 2.4 (ships with Spark 1.6 and Kafka 0.9) HDF 2.1 (always latest NiFi) Encryption & Kerberos in and out of Datalake. Not one tutorial on the web cares about security Constraints
- 12. TELUS Public 12 ® First Attempt - Kafka Set Top Boxes Actions / Alerts Insights / Analytics Hadoop network
- 13. TELUS Public 13 Kafka 0.9 introduced SSL/TLS, but Spark 1.6 uses the Kafka 0.8 client library, so can’t do SSL. Solved by using Spark 2.x, but: NiFi can only use one Kerberos principal Kafka, thwarted!
- 14. TELUS Public 14 Site to Site (S2S) provides two way SSL between NiFi instances. SparkStreaming can stream using S2S protocol. Careful… No easy way to limit NiFi resources by user/group. Multiple NiFi?
- 15. TELUS Public 15 NiFi outside the Datalake does the heavy work to process incoming STB logs. NiFi within the Datalake does the lighter work: – Store in HDFS – Feeds Spark Streaming using S2S via 2-way SSL – Lighter load improves multi-tenancy (scale) for NiFi Second Attempt – NiFi Site to Site
- 16. TELUS Public 16 Final Architecture – NiFi S2S Set Top Boxes Actions & Alerts Insights & Analytics Hadoop network ML results Store ®
- 17. TELUS Public 17 No prior labels to demonstrate a failing STB Unsupervised classifier as first pass SparkML using KMeans to build model Problem Detection
- 18. TELUS Public 18 Model Building
- 19. TELUS Public 19 Lessons Learned
- 20. TELUS Public 20 Java 8 everywhere – SSL ‘fun’ with J7 vs J8 – NiFi >= 1.0 libraries compiled with J8 At least Spark 2.0, ideally 2.1 or newer – SSL/TLS with Kafka – Speed – Improved SparkML – Structured Streaming Lessons Learned
- 21. TELUS Public 21 NiFi Kafka support one Kerberos principal per NiFi server. NiFi HDFS supports multiple Kerberos principals, however, server process can read all keytabs. More Lessons Learned
- 22. TELUS Public 22 NiFi site to site squashes identity to that of the sending server. Get good at SSL and how certificates work. Highly secured multi-tenancy is hard! More Lessons Learned
- 23. TELUS Public 23 Cavan Loughran, TELUS, www.telus.com Oliver Meyn, T4G, www.t4g.com Thank you!
- 24. TELUS Public 24 the future is friendly.