Cassandra + Hadoop = Brisk
Download as PPTX, PDF7 likes11,371 views
The document discusses a platform called Brisk, which integrates Hadoop, Hive, and Cassandra to manage data for an ad network targeting user segmentation and real-time analytics. It includes technical details about the data model, setup instructions, and PHP code examples for implementing user tracking via a pixel-based system. The document emphasizes ease of use, mixed-mode querying, and the absence of a single point of failure in the architecture.
![Data modelCF = users[userUUID] [segmentID] = 1CF = segments[segmentID] [userUUID] = 1](https://image.slidesharecdn.com/brisktalk-110517055123-phpapp01/85/Cassandra-Hadoop-Brisk-23-320.jpg)
![Data modelcreate keyspacewhyk... with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' ... and strategy_options = [{replication_factor:1}];create column family users ... with comparator = 'AsciiType'... and rows_cached = 5000;create column family segments... with comparator = 'AsciiType'... and rows_cached = 5000;](https://image.slidesharecdn.com/brisktalk-110517055123-phpapp01/85/Cassandra-Hadoop-Brisk-24-320.jpg)
![Data modelcreate keyspacewhyk... with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' ... and strategy_options = [{replication_factor:1}];create column family users ... with comparator = 'AsciiType'... and rows_cached = 5000;create column family segments... with comparator = 'AsciiType'... and rows_cached = 5000;](https://image.slidesharecdn.com/brisktalk-110517055123-phpapp01/85/Cassandra-Hadoop-Brisk-25-320.jpg)
Cassandra + Hadoop = Brisk
- 2. But first, a short back story…
- 7. Please volunteer if you would like to give a talk, Internet fame awaits
- 8. My experience with Cassandra in production is positive
- 9. Analytics is more difficult than it could be
- 10. Welcome Brisk! Brisk combines Hadoop, Hive and Cassandra in a “distribution”
- 11. In a nutshellCassandraFS as HDFS compatible layer; no namenode, no SPOF
- 12. Can split cluster for OLAP and OLTP workloads, scaling up either as requiredDemonstrating brisk…Building an Ad Network!
- 13. Demonstrating brisk…Building anAd Network!
- 14. The plan: Simple data model – segment users into buckets
- 15. System to put users in buckets via a pixel
- 17. AnalyticsWe Have Your KidneysThe ad-network for the paranoid generation Cookie based identification
- 18. API provides:
- 19. Add user to a bucket (including ability to define expiry time)
- 20. Get buckets a user belongs toSetup Briskhttp://www.datastax.com/docs/0.8/brisk/install_brisk_ami Step-by-step guide with pictures!
- 21. Ubuntu 10.10 image with RAID 0 ephemeral disks
- 22. Jairam has been bug-fixing some minor issues
- 23. Data modelCF = users[userUUID] [segmentID] = 1CF = segments[segmentID] [userUUID] = 1
- 24. Data modelcreate keyspacewhyk... with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' ... and strategy_options = [{replication_factor:1}];create column family users ... with comparator = 'AsciiType'... and rows_cached = 5000;create column family segments... with comparator = 'AsciiType'... and rows_cached = 5000;
- 25. Data modelcreate keyspacewhyk... with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' ... and strategy_options = [{replication_factor:1}];create column family users ... with comparator = 'AsciiType'... and rows_cached = 5000;create column family segments... with comparator = 'AsciiType'... and rows_cached = 5000;
- 26. Our pixelhttp://wehaveyourkidneys.com/add.php? segment=<alphaNumericCode> &expire=<numberOfSeconds> We’ll use Cassandra’s expiring columns feature PHP code – uses phpcassa$pool = new ConnectionPool('whyk', array('localhost'));$users = new ColumnFamily($pool, 'users');$segments = new ColumnFamily($pool, 'segments');$users->insert($userUuid,array($segment => 1),NULL, // default TS$expires );$segments->insert($segment,array($userUuid => 1),NULL, // default TS$expires );
- 27. Real-time accesshttp://wehaveyourkidneys.com/show.php$pool = new ConnectionPool('whyk', array('localhost'));$users = new ColumnFamily($pool, 'users');// @todo this only gets first 100!$segments = $users->get($userUuid);header('Content-Type: application/json');echo json_encode(array_keys($segments));
- 28. AnalyticsHow many users in each segment?Launch HIVE (very easy!)root@brisk-01:~# brisk hive
- 29. CREATE EXTERNAL TABLE whyk.users (userUuid string, segmentId string, value string)STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler’WITH SERDEPROPERTIES ("cassandra.columns.mapping" = ":key,:column,:value" );select segmentId, count(1) as totalfrom whyk.usersgroup by segmentIdorder by total desc;
- 31. Real time access+Batch analytics
- 32. EasyEasy to setupEasy to deploy mixed-modeclustersEasy to query (Hive)
- 33. No Single Pointof Failure
- 34. Further reading…Installing the Brisk AMIhttp://www.datastax.com/docs/0.8/brisk/install_brisk_amiKey advantages of Brisk – from Jonathan Ellishttp://hackerne.ws/item?id=2528271Why I’m very excited about DataStax’s Brisk – by Nathan Milfordhttp://blog.milford.io/2011/04/why-i-am-very-excited-about-datastaxs-brisk/The demo code on Githubhttps://github.com/davegardnerisme/we-have-your-kidneys
Editor's Notes
- #4: Started at Imagini; May 2010New ad-targeting product! Lots of users.MySQL DB for profiles, MySQL based server for events reportingProfile DB cannot update rows so we only insert; this means clients have to merge together all rows for a user on every readMySQL DB has a habbit of dying, requiring a repair and downtime; having 2 DBs managed to put off total death but not for long
- #5: Choosing Cassandra after some research; no single point of failure attractive, high write throughput attractive, linear scaling attractiveWelcome to GC hell!Start Cassandra London – like alcoholics anonymous; a support network
- #6: Batch analytics; how? No Hive support, no support for streaming jarPig input readerNo output reader; require HDFS
- #7: Keep up the meetupsAcunu generous at providing speakers; downside is hearing sales pitch!0.7 comes along; downside is not compatible with 0.6; Thrift interface changes0.8 comes along; CQL, countersBrisk!
- #9: A summary
- #10: Some points about “distribution” Some points about Cloudera and reaction
- #11: Realtime + batch analytics combinedNo single point of failure; we don’t need Hadoop’snamenode anymoreCross DC clusters
- #14: No adsNo networkNo publishersCool domain name
- #15: User segments / buckets; have some ID, can expire (we’ll use Cassandra’s expiring columns)Real-time updates via a simple script (via CQL?)Real-time queries (is user in this segment? What segments is user X in?)Analytics (how many users in each segment? How many segments are users in on average? Std dev?)
- #16: User segments / buckets; have some ID, can expire (we’ll use Cassandra’s expiring columns)Real-time updates via a simple script (via CQL?)Real-time queries (is user in this segment? What segments is user X in?)Analytics (how many users in each segment? How many segments are users in on average? Std dev?)
- #22: User segments / buckets; have some ID, can expire (we’ll use Cassandra’s expiring columns)Real-time updates via a simple script (via CQL?)Real-time queries (is user in this segment? What segments is user X in?)Analytics (how many users in each segment? How many segments are users in on average? Std dev?)
- #23: User segments / buckets; have some ID, can expire (we’ll use Cassandra’s expiring columns)Real-time updates via a simple script (via CQL?)Real-time queries (is user in this segment? What segments is user X in?)Analytics (how many users in each segment? How many segments are users in on average? Std dev?)