Hadoop and Storm - AJUG talk
- 1. ©MapR Technologies Hadoop and Storm AJUG 5/21/2013
- 2. whoami • Brad Anderson • Solutions Architect at MapR (Atlanta) • ATLHUG co-chair • NoSQL East Conference 2009 • “boorad” most places (twitter, github) • banderson@maprtech.com
- 3. Hadoop: A Paradigm Shift Distributed computing platform – Large clusters – Commodity hardware Pioneered at Google – Google File System, MapReduce and BigTable Commercially available as Hadoop
- 4. Ship the Function to the Data SAN/NAS data data data data data data data data data data data data data data data function RDBMS Traditional Architecture data function data function data function data function data function data function data function data function data function data function data function data function Distributed Computing
- 5. MapReduce Flow Input Map Combin e Shuffle and sort Reduc e Output Reduc e
- 6. Variation: No Reduce Necessary Example: Batch File Transformation Input Map Output MPG M4V
- 7. Variation: Multiple MapReduces Example: Fraud Detection in User Transactions LDA training Transaction data LDA scoring HBase / MapR M7 Edition G2 score Candidate events for analyst review 95 %-ile LDA anomaly MapReduce http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation
- 8. Pig
- 9. MR Equivalent to Pig Script
- 10. Hive
- 11. MapR Distribution for Apache Hadoop Complete Hadoop distribution Comprehensive management suite Industry-standard interfaces Enterprise-grade dependability Enterprise-grade security (US Intelligence Agency) Patents - IP Higher performance
- 12. Hadoop Use Cases ETL/EDW Offload Sensor / Telemetry Data Recommendation Engine Search •ML algorithms •eDiscovery Fleet Management Fraud Detection / Risk Management Traffic Decongestion
- 13. One Platform for Big Data … 99.999% HA Data Protection Disaster Recovery Scalability & Performance Enterprise Integration Multi- tenancy Map Reduce File-Based Applications SQL Database Search Stream Processing Batc h Interactiv e Realtime Batch Log file Analysis Data Warehouse Offload Fraud Detection Clickstream Analytics Realtime Sensor Analysis “Twitterscraping” Telematics Process Optimization Interactive Forensic Analysis Analytic Modeling BI User Focus
- 14. ©MapR Technologies Storm “Hadoop for Realtime”
- 15. ©MapR Technologies Before Storm Queues Workers
- 17. ©MapR Technologies Storm Guaranteed data processing Horizontal scalability Fault-tolerance No intermediate message brokers! Higher level abstraction than message passing “Just works”
- 18. ©MapR Technologies Unbounded sequence of tuples Tuple Tuple Tuple Tuple Tuple Tuple Tuple Streams
- 19. ©MapR Technologies Source of streams Spouts
- 20. ©MapR Technologies public interface ISpout extends Serializable { void open(Map conf, TopologyContext context, SpoutOutputCollector collector); void close(); void nextTuple(); void ack(Object msgId); void fail(Object msgId); } Spouts
- 21. ©MapR Technologies Processes input streams and produces new streams Tuple Tuple Tuple Tuple Bolts
- 22. ©MapR Technologies public class DoubleAndTripleBolt extends BaseRichBolt { private OutputCollectorBase _collector; public void prepare(Map conf, TopologyContext context, OutputCollectorBase collector) { _collector = collector; } public void execute(Tuple input) { int val = input.getInteger(0); _collector.emit(input, new Values(val*2, val*3)); _collector.ack(input); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("double", "triple")); } } Bolts
- 23. ©MapR Technologies Network of spouts and bolts Topologies
- 24. ©MapR Technologies TridentTopology topology = new TridentTopology(); TridentState wordCounts = topology.newStream("spout1", spout) .each(new Fields("sentence"), new Split(), new Fields("word")) .groupBy(new Fields("word")) .persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count")) .parallelismHint(6); Trident Cascading for Storm
- 27. StormKafka Twitter Twitter API TweetLogger Kafka Cluster Kafka Cluster Kafka Cluster Kafka API Storm Web Service NAS Web Data Hadoop Flume HDFS Data
- 29. Scaling Estimates Twitter Firehose Old School – 8+ separate clusters, 20-25 nodes • >3 Kafka nodes • >2 TweetLoggers • 5-10 Hadoop • >2 Catcher nodes • >3 Storm • 3 zookeepers • NAS for web storage • >2 web servers MapR – One Platform • 5-10 nodes total • Any node does any job • Full HA included • Backups included
- 30. ©MapR Technologies github • Watch TailSpout & Franz development • https://github.com/{tdunning | boorad | pfcurtis}/mapr-spout • And our example Twitter implementation • https://github.com/{tdunning | boorad | pfcurtis}/mapr-spout-test
- 31. Demo