HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil
Download as PPTX, PDF1 like3,359 views
The document discusses the development of real-time analytics applications using HBase in the cloud for brand protection against counterfeiting. It emphasizes the importance of schema design and efficient data processing, highlighting that traditional MapReduce is too slow for real-time needs. The paper also points out best practices, such as careful use of filters and effective row key packing, to optimize HBase implementation.
HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil
- 1. Developing Real Time Analytics Applications Using HBase in the Cloud May 22, 2012 Rick Tucker tech@sproxil.com tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.
- 2. About Sproxil • Brand protection, specializing in anti- 1 SCRATCH counterfeiting solutions • Solution requires a scalable and high- throughput text 2 message processing TEXT engine • Supports a real-time analytics web interface 3 VERIFY tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.
- 3. Why HBase? USER SENDS TEXT MESSAGE CALCULATE TEXT MESSAGE IS PROCESSED ANALYTICS USER Amazon EC2 RECEIVES Cloud REPLY tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.
- 4. Real-Time Analytics Engine • MapReduce too slow to maintain data in true real time • As data arrives, analytical data is updated through counters Text Message Message Increment Arrives Analyzed Counters Genuine Product +1 Increment Counter for Authentication Genuine Authentications Repeat Customer +1 Increment Counter for Repeat Customers tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.
- 5. Schema Design: Example 1 • Example: View log of text messages in chronological order • Rowkey: row prefix + timestamp Row transaction 2012-05-22 12:00:00 transaction 2012-05-22 12:01:14 transaction 2012-05-22 12:02:03 Note: HBase sorts rowkeys lexicographically so scans return data in reverse chronological order tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc. 5
- 6. • • Row transaction userID 1 2012-05-22 12:00:00 transaction userID 1 2012-05-22 12:01:14 transaction userID 2 2012-05-22 12:00:54 transaction userID 2 2012-05-22 12:01:22 transaction userID 2 2012-05-22 12:02:01 Note: Hbase sorts rows lexicographically so scans return data in reverse chronological order tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.
- 7. Critical Findings • Schema design is crucial for successful HBase implementation – Pack as much info as possible into row keys • Use caution with Filters – E.g. Regex filters can be costly – Alternatives: • Directly query for data you need • Use efficient filters when filtering large data sets tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.
- 8. Thank You! Your global brand protection specialists – spanning 3 continents and Making Counterfeiting Unprofitable™ speaking 9 languages tech@sproxil.com +1 617 682 9577 America | Asia | Africa Sproxil.com tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc. 8