Hortonworks HDP, Is it goog enough ?
8 likes650 views
The document discusses the capabilities and advantages of the Hadoop Big Data platform, particularly focusing on Hortonworks Data Platform (HDP) as an enterprise-grade solution. It emphasizes Hadoop's design for big data processing, its scalability, mature ecosystem, and built-in security features, while highlighting the digital trends driving its adoption. Additionally, it covers the deployment of Hadoop clusters using Ambari and showcases typical use cases through demos.
Hortonworks HDP, Is it goog enough ?
- 1. © Huxi LI, 2018 HADOOP BIG DATA PLATFORM Hortonworks HDP, Is it good enough ?
- 2. © Huxi LI, 2018 AGENDA 1|Why Hadoop ? 2|Hortonworks HDP, is it good enough ? 3|Hadoop cluster provision with Ambari 4|Demos of typical use cases Annexes 2
- 3. © Huxi LI, 2018 Digitalization INDUSTRIAL TRENDS • Digitalization of core business: Going digital is now a core strategy for many organizations around the world, including financial et insurance core business. Most successful enterprises deliver their core business through digital channels. • Digitalization of society: Social networks, collaborative platforms, e- government, and smart devices are changing the entire society. - Social network is everywhere and start impacting the business model; - Industrialized countries are pushing the digitalization of their entire public services as a way of modernizing governments and better serving their citizens. • Internet Of Thing (IoT): The internet are expanding to physical objects; experts estimate that the IoT will consist of about 30 billion objects by 2020 (IEEE, 2016). • Artificial Intelligence (AI): The maturity of AI reaches another level, from academic experimental stage into massive practical usage era. AI not only need massive data and but also produce massive data. 3 A matter of Data
- 4. © Huxi LI, 2018 AGE OF BIG DATA 4 Big Volume High Variety Uncertain Veracity High Velocity In 2016, global mobile data traffic amounted to 7 exabytes per month. In 2021, mobile data traffic worldwide is expected to reach 49 exabytes per month at a compound annual growth rate of 47 percent (Statista, 2018) Variety of data can be numerous - structured, semi-structured and mostly unstructured data as well. Velocity of data producing can be variable from slow batch processing, near real-time streaming, to real-time IoT sensors. Veracity is one of the unfortunate characteristics of today’s data. Not all data are trustworthy.
- 5. © Huxi LI, 2018 HADOOP = DESIGNED FOR BIG DATA Hadoop allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up to thousands of machines, each offering local computation and storage. • Massive Storage (Volume): Hadoop’s distributed file system (HDFS), is specially designed to store massive data on commodity hardware, providing high-throughput access to application data. • Any kind of Data (Variety): You can save text, image, video, XML, JSON and any other type of data in HDFS, as if you are working on a unix-like file system - this is very important because of the variety of today’s data. • Massively Scalable Computation: Parallel computing is integrated into Hadoop core system – YARN for cluster resource management and MapReduce for parallel processing of large data sets. And many other derived tools such as Storm, Hive, and so on you can choose from. 5
- 6. © Huxi LI, 2018 HADOOP = A MATURE SYSTEM • Production-proved: Hadoop has been confirmed in production by many companies around the world. • Available on Cloud: Hadoop are now deployed on all major cloud platforms such Microsoft AZURE, Amazon AWS, IBM Cloud, and Google GCP. 6
- 7. © Huxi LI, 2018 HADOOP = A RICH ECOSYSTEM The term Hadoop refers to the entire ecosystem or collection of software packages that can be installed on top of or alongside Hadoop. Many choices : • Ingestion : Scoop, Flume, Kafka, NIFI, Storm, Flink, … • Streaming & Compute: MapReduce, Storm, Spark, Flink, Nifi, … • Analysis/SQL-on-Hadoop: Impala, Hive, Drill, … • Machine learning: Mahout, Spark ML, … • NoSQL: HBase, Cassandra, … • Management: Ambari, HUE, … 7
- 8. © Huxi LI, 2018 VALUE PROPOSITIONS OF HADOOP 8 Designed for Big Data • Massively scalable storage and computation • Carefully designed for large scale deployment High Maturity • Production-proved, highly robust • Confirmed by big clusters with thousands of nodes Rich Ecosystem • Ecosystem formed around Hadoop with many choices, with enterprise-grade quality but freely available • Wide supports, commercially or of community. High Scalability • Technically, a single Hadoop cluster can be scaled to thousands of nodes • Functionally, Hadoop support any kind of data, scalable to a wide range of use cases.
- 9. © Huxi LI, 2018 AGENDA 1|Why Hadoop ? 2|Hortonworks HDP, is it good enough ? 3|Hadoop cluster provision with Ambari 4|Demos of typical use cases Annexes 9
- 10. © Huxi LI, 2018 “Enterprises like Hortonworks’ storage and compute processing, broad data ingestion, data governance, and open source support when deploying BDW.” - Forrester 2017 HDP = ENTERPRISE-GRADE HADOOP + 100% OPEN SOURCE Enterprise-grade Hadoop Hortonworks HDP is the only Enterprise-Grade Hadoop distribution of fully Open Source Apache License All of the technology built into the Hortonworks Data Platform is an Apache open source project 100% Open Source Hortonworks HDP is the only Enterprise-Grade Hadoop distribution of fully Open Source No Vendor Lock-In HDP delivers enterprise-grade software that fosters innovation and prevents vendor lock-in HDP
- 11. © Huxi LI, 2018 HDP = CAREFULLY ENGINEERED SECURITY ARCHITECTURE • Platform built-in security and data governance. Built-In Security • Support integration into enterprise’s existing security system with support of industrial standards, such as Kerberos and LDAP. Corporate Integration • Centralized cluster management, monitoring, and access control. Centralized control • Simple yet powerful, Unix-like, resource permission control (HDFS) Simplicity
- 12. © Huxi LI, 2018 HDP = LEADER OF BIG DATA WAREHOUSE • Hortonworks was ranked as a leader in The Forrester Wave™: Big Data Warehouse (Q2 2017). 12 “Hortonworks delivers actionable intelligence from all kinds of data-in-motion and data-at-rest. Through its open source strategy, Hortonworks continually evolves its offering by working closely with partners across the EDW ecosystem of tools and vendors. The vendor provides a cost-effective, nimble, and scalable architecture to implement big data warehouses, whether on-premises or in the cloud.” - Forrester 2017
- 13. © Huxi LI, 2018 HORTONWORKS HDP 13
- 14. © Huxi LI, 2018 VALUE PROPOSITIONS OF HORTONWORKS HDP Cost • No license fee • Pay for support Openness • Open source • With commercial friendly Apache license Expertise • Key contributors of Hadoop • Decade of industrial Big Data experience Security • Guarantee of a market leader • Mature technology, Support on premise, cloud, or appliance.
- 15. © Huxi LI, 2018 AGENDA 1|Why Hadoop ? 2|Hortonworks HDP, is it good for me ? 3|Hadoop Cluster Provision with Ambari 4|Demos of typical use cases Annexes 15
- 16. © Huxi LI, 2018 AMBARI HADOOP CLUSTER PROVISION Explore Hadoop Cluster With Ambari Create Hadoop Cluster With Ambari Install Ambari server Prepare VMs Ambari 1 2 3 4 Manual, Ansible, Puppet …
- 17. © Huxi LI, 2018 TYPICAL HADOOP CLUSTERS • Hadoop backbone: The Hadoop backbone is typically composed of HDFS, MapReduce and YARN, i.e. the core of Hadoop cluster: • HDFS for massive storage service • MapReduce for parallel computing of datasets • YARN for job scheduling or resource Mgt • Edge nodes: Edge nodes are part of the Hadoop cluster, but they are typically connected to the rest of the corporate network, providing operational environments to end users. • Ambari is a typical edge node providing cluster management service • Scoop & Flume are another kind of edge nodes providing dataflow IN or OUT of the cluster • Spark & Hive are edge nodes serving applications or business users 17 Corporate Network Administration Applications Edge nodes (e.g. Hive) Management (e.g. Ambari)
- 18. © Huxi LI, 2018 A PROOF-OF-CONCEPT HADOOP CLUSTER • POC Hadoop cluster: • Composed of 3 nodes, • Share VMs for edge and backbone nodes • Hortonworks HDP 2.6.1.3 • AWS VPC 172.31.0.0/16 • Provisioner: • Ambari as provisioner of the cluster 18 VPC: 172.31.0.0/16 hdp-master (172.31.45.112) hdp-slave1 (172.31.37.152) hdp-slave2 (172.31.45.181) Namenode, Ambari, etc. Datanode, Hive, Storm, etc. Datanode, Nodemanager, etc.
- 19. © Huxi LI, 2018 MACHINE SETUP • Provision VMs with AWS: • 3 EC2 instances • Redhat 7.0 Linux • 1 VPC allowing traffic among nodes 19
- 20. © Huxi LI, 2018 NETWORK TRAFFIC SETUP • Authorized traffic (AWS): 20 Cluster internal traffic
- 21. © Huxi LI, 2018 INSTALL AMBARI SERVER • Follow the official installation guide for details 21
- 22. © Huxi LI, 2018 SETUP HADOOP CLUSTER WITH AMBARI 22 Click Here Ambari Web UI: http://hdp-master.huxili.com:8080
- 23. © Huxi LI, 2018 SETUP HADOOP CLUSTER WITH AMBARI 23 Choose Hadoop nodes
- 24. © Huxi LI, 2018 EXPLORE HADOOP CLUSTER WITH AMBARI 24
- 25. © Huxi LI, 2018 AGENDA 1|Why Hadoop ? 2|Hortonworks HDP, is it good for me ? 3|Hadoop Cluster Provision with Ambari 4|Demos of typical use cases Annexes 25
- 26. © Huxi LI, 201826 1. Data storage in HDFS …
- 27. © Huxi LI, 2018 DEMO 1 - SAVING DATA IN HDFS Ambari UI Upload Files Create directory File View • Open Ambari UI at http://hdp-master.huxili.com Manage with Ambari • Open File View from Ambari UI • Create a directory in HDFS : /user/huxili/dfs_demo • Manipulate uploaded files (rename, permission, Ctrl+Mouse => Select directory) Set permission • Upload files into the above create directory
- 28. © Huxi LI, 201828 2. JSON Data query with Hive …
- 29. © Huxi LI, 2018 DEMO 2 – DATA QUERY WITH HIVE Ambari UI Create table for JSON data Create schema for DEMO Hive View • Check sample twitter data (JSON) in /user/huxili/twitter_data Hive Query with Ambari • Open Ambari Hive View 2.0 • Create a database for demo purpose -> create database demo; • Query JSON data in Hive Query JSON Data in Hive • Create an external table demo_tweets pointing to /user/huxili/twitter_data
- 30. © Huxi LI, 201830 3. Data Streaming with Storm …
- 31. © Huxi LI, 2018 DEMO 3 – STREAMING OF TWEETS WITH STORM 31 StreamingSpout Tweets Splitter Bolt Filter Bolt Counter Bolt Aggregation & HDFS writing Bolt Hadoop HDFS (File: /user/huxili/storm_out/counts.txt)
- 32. © Huxi LI, 2018 DEMO 3 – DATA STREAMING WITH STORM Ambari UI SSH into Storm Host Upload Streaming Example File View • Open Ambari File View -> Upload example streaming jar Explore Results in Ambari • SSH into hdp-slave1 (Storm host) • su root & download the streaming example (hdfs dfs -get /user/huxili/storm_apps/tw*.jar) • Streaming will stop after 30 seconds and explore results in Ambari Submit streaming App to Storm • Submit the streaming to storm
- 33. © Huxi LI, 201833 Annexes
- 34. © Huxi LI, 2018 ANNEXES A1|Ambari Server Setup • Provision VMs on AWS • Install required packages • Install Ambari Server A2|Hadoop Cluster Setup A3|Streaming Architecture Hive/Storm 34
- 35. © Huxi LI, 2018 • Provision VMs: • Using AWS EC2 instance • Redhat 7.0 Linux • AWS / VPC, 3 nodes A1 - AMBARI SERVER SETUP 35 MACHINES SETUP
- 36. © Huxi LI, 2018 • Updating /etc/hosts (All nodes): • 172.31.45.112 hdp-master.huxili.com • 172.31.37.152 hdp-slave1.huxili.com • 172.31.45.181 hdp-slave2.huxili.com • Updating /etc/hostname (All node): • hdp-master.huxili.com (Master) • hdp-slave1.huxili.com (slave1) • hdp-slave2.huxili.com (slave2) A1 - AMBARI SERVER SETUP 36 MACHINES SETUP
- 37. © Huxi LI, 2018 • Authorized traffic (AWS): A1 - AMBARI SERVER SETUP 37 NETWORK TRAFFIC SETUP Cluster internal traffic
- 38. © Huxi LI, 2018 ANNEXES A1|Ambari Server Setup • Provision VMs on AWS • Install required packages • Install Ambari Server A2|Hadoop Cluster Setup A3|Streaming Architecture Hive/Storm 38
- 39. © Huxi LI, 2018 • Hdp-master.huxli.com: • sudo yum install -y ntp • sudo systemctl enable ntpd • sudo yum install -y postgresql-jdbc • sudo yum install -y wget • sudo wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.3/ambari.repo -O /etc/yum.repos.d/ambari.repo • sudo yum install -y ambari-server A1 - AMBARI SERVER SETUP 39 INSTALL REQUIRED PACKAGES
- 40. © Huxi LI, 2018 • hdp-slave1.huxili.com & hdp-slave2.huxili.com: • sudo yum install -y ntp • sudo systemctl enable ntpd A1 - AMBARI SERVER SETUP 40 INSTALL REQUIRED PACKAGES
- 41. © Huxi LI, 2018 A1 - AMBARI SERVER SETUP 41 INSTALL REQUIRED PACKAGES • hdp-slave1.huxili.com: Install hive metastore DB (postgres)
- 42. © Huxi LI, 2018 A1 - AMBARI SERVER SETUP 42 INSTALL REQUIRED PACKAGES • hdp-slave1.huxili.com: Check Hive DB
- 43. © Huxi LI, 2018 ANNEXES A1|Ambari Server Setup • Provision VMs on AWS • Install required packages • Install Ambari Server A2|Setup Hadoop Cluster A3|Streaming Architecture Hive/Storm 43
- 44. © Huxi LI, 2018 • Commands to run: A1 - AMBARI SERVER SETUP 44 INSTALL SERVER
- 45. © Huxi LI, 2018 • Configure JDBC driver: • sudo ambari-server setup --jdbc-db=postgres --jdbc-driver=/usr/share/java/postgresql-jdbc.jar A1 - AMBARI SERVER SETUP 45 INSTALL SERVER
- 46. © Huxi LI, 2018 • Setup server: • sudo ambary-server setup A1 - AMBARI SERVER SETUP 46 INSTALL SERVER
- 47. © Huxi LI, 2018 • Setup server (continued): A1 - AMBARI SERVER SETUP 47 INSTALL SERVER
- 48. © Huxi LI, 2018 • Start server: A1 - AMBARI SERVER SETUP 48 INSTALL SERVER
- 49. © Huxi LI, 2018 • Check services binding: A1 - AMBARI SERVER SETUP 49 INSTALL SERVER
- 50. © Huxi LI, 2018 ANNEXES A1|Ambari Server Setup A2|Hadoop Cluster Setup • Install Hadoop components • Check cluster sanity A3|Streaming Architecture Hive/Storm 50
- 51. © Huxi LI, 2018 • Open Ambari management UI: • http://hdp-master.huxili.com:8080 • User: admin • Pwd: admin A2 - HADOOP CLUSTER SETUP 51 STEP 1 – OPEN MANAGEMENT UI
- 52. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 52 STEP 2 – START CREATION WIZARD Click Here
- 53. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 53 STEP 3 – CHOOSE CLUSTER NAME
- 54. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 54 STEP 4 – CHOOSE VERSION
- 55. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 55 STEP 5 – CHOOSE PARAMETERS
- 56. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 56 STEP 6 – AMBARI HOST CHECK (FAILED) Error
- 57. © Huxi LI, 2018 • vi /etc/hostname and using the right hostname • hdp-master.huxili.com • hdp-slave1.huxili.com • hdp-slave2.huxili.com A2 - HADOOP CLUSTER SETUP 57 STEP 6 – UPDATE HOSTNAMES AND RETRY
- 58. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 58 STEP 7 – CHOOSE REQUIRED SERVICES
- 59. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 59 STEP 8 – DISTRIBUTION OF COMPONENTS
- 60. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 60 STEP 8 – DISTRIBUTION OF COMPONENTS
- 61. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 61 STEP 9 – CONFIGURE HIVE DB (POSTGRES)
- 62. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 62 STEP 10 – CONTINUED
- 63. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 63 STEP 11 – CONTINUED
- 64. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 64 STEP 12 – CONTINUED
- 65. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 65 STEP 13 – CONTINUED
- 66. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 66 STEP 14 – INSTALLATION FAILED
- 67. © Huxi LI, 2018 • Error messages : • Solutions : manually install missing package • sudo yum-config-manager --enable rhui-REGION-rhel-server-optional • sudo yum install libtirpc-devel A2 - HADOOP CLUSTER SETUP 67 STEP 15 – ANALYZE ERROR MESSAGE Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_6_4_0_91-client' returned 1. Error: Package: hadoop_2_6_4_0_91-hdfs-2.7.3.2.6.4.0-91.x86_64 (HDP-2.6-repo-1) Requires: libtirpc-devel You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
- 68. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 68 STEP 16 – INSTALL MISSING PACKAGES
- 69. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 69 STEP 17 – INSTALL SUCCESS
- 70. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 70 STEP 18 – AMBARI DASHBOARD
- 71. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 71 STEP 19 – CREATE USER ‘HUXILI’
- 72. © Huxi LI, 2018 ANNEXES A1|Ambari Server Setup A2|Hadoop Cluster Setup • Install Hadoop components • Check cluster sanity A3|Streaming Architecture Hive/Storm 72
- 73. © Huxi LI, 2018 • Open HDFS File View failed: A2 - HADOOP CLUSTER SETUP 73 CHECK FILE VIEW
- 74. © Huxi LI, 2018 • Check view definitions (cluster name is empty): A2 - HADOOP CLUSTER SETUP 74 CHECK FILE VIEW
- 75. © Huxi LI, 2018 • HDFS File View after filling cluster name: A2 - HADOOP CLUSTER SETUP 75 CHECK FILE VIEW
- 76. © Huxi LI, 2018 • Open Hive View failed: • Missing ‘/user/huxili’ A2 - HADOOP CLUSTER SETUP 76 CHECK HIVE VIEW
- 77. © Huxi LI, 2018 • Create directory for custom Hive user: • hdfs dfs -ls / && hdfs dfs -ls /user A2 - HADOOP CLUSTER SETUP 77 CHECK HIVE VIEW
- 78. © Huxi LI, 2018 • Check HDFS: • hdfs dfs -getfacl /user • Create /user/huxili & admin • su hdfs • hdfs dfs -mkdir /user/huxili & hdfs dfs -chown huxili /user/huxili • hdfs dfs -mkdir /user/admin & hdfs dfs -chown admin /user/admin A2 - HADOOP CLUSTER SETUP 78 CHECK HIVE VIEW
- 79. © Huxi LI, 2018 • Open Hive View OK after creation of missing directory: A2 - HADOOP CLUSTER SETUP 79 CHECK HIVE VIEW
- 80. © Huxi LI, 2018 • Open Hive View 2.0 (OK) : A2 - HADOOP CLUSTER SETUP 80 CHECK HIVE VIEW 2.0
- 81. © Huxi LI, 2018 • Check service binding: • netstat -tulpn A2 - HADOOP CLUSTER SETUP 81 CHECK SERVICE BINDING
- 82. © Huxi LI, 2018 • Service binding problems: • Zookeeper listening only on IP6 and clustering join failed : • Log error messages: • (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server hdp-slave1.huxili.coms/... Will not attempt to authenticate using SASL (unknown error) • Solution: • Force zookeeper use IP4 by adding the following line in “Advanced zookeeper-env => template”: ‐ export SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djava.net.preferIPv4Stack=true" A2 - HADOOP CLUSTER SETUP 82 CHECK SERVICE BINDING
- 83. © Huxi LI, 2018 • Force IP4 for the following services: • ambari-infra-solr : • SOLR_OPTS="$SOLR_OPTS -Djava.net.preferIPv4Stack=true“ • SmartSense: • export ANALYZER_JAVA_OPTS="{{analyzer_jvm_opts}} -Djava.net.preferIPv4Stack=true -Xmx{{analyzer_jvm_heap}}m“ • export ZEPPELIN_JAVA_OPTS="-Dhdp.version={{hdp_version}} -Dlog.file.name=activity-explorer.log - DSmartSenseActivityExplorer -Djava.net.preferIPv4Stack=true“ • YARN: • YARN_OPTS="$YARN_OPTS -Djava.net.preferIPv4Stack=true“ • STORM: • DRPC: -Xmx768m _JAAS_PLACEHOLDER -Djava.net.preferIPv4Stack=true • Numbus: -Xmx1024m _JAAS_PLACEHOLDER -Djava.net.preferIPv4Stack=true • Supervisor & UI & Storm-site/logviewer: -Djava.net.preferIPv4Stack=true A2 - HADOOP CLUSTER SETUP 83 CHECK SERVICE BINDING
- 84. © Huxi LI, 2018 A2 - HADOOP CLUSTER SETUP 84 SERVICE INSTALL SUCCESSFUL !
- 85. © Huxi LI, 2018 ANNEXES A1|Ambari Server A2|Hadoop Cluster Setup A3|Streaming Architecture Hive/Storm • Store twitter data in HDFS • Hive Query of Twitter data • Twitter streaming with Storm 85
- 86. © Huxi LI, 2018 • Create account huxili using Ambari • Create directories in HDFS : • Login as hdfs (su hdfs) • Create /user/huxili • Change owner to huxili • Now login to ambari as user ‘huxili’ and open ‘Files view’ from the menu. Since huxili is the owner of the newly created directory, we can add data into this directory. I create the following directory to save Twitter data: • /user/huxili/twitter_data A3 - STREAMING ARCHITECTURE HIVE/STORM 86 STORE TWITTER DATA
- 87. © Huxi LI, 2018 • Open Hive view 2.0 and execute: • CREATE DATABASE `huxili` • Create Table ‘tweets’ : SET hive.support.sql11.reserved.keywords=false; DROP TABLE IF EXISTS tweets; CREATE EXTERNAL TABLE tweets (createddate string, geolocation string, tweetmessage string, `user` struct<geoenabled:boolean, id:int, name:string, screenname:string, userlocation:string>) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' STORED AS TEXTFILE LOCATION '/user/huxili/twitter_data/' A3 - STREAMING ARCHITECTURE HIVE/STORM 87 HIVE QUERY OF TWITTER DATA – CREATE TABLE
- 88. © Huxi LI, 2018 • Create Table ‘tweets’ OK: A3 - STREAMING ARCHITECTURE HIVE/STORM 88 HIVE QUERY OF TWITTER DATA – CREATE TABLE CONTINUED
- 89. © Huxi LI, 2018 • SET hive.support.sql11.reserved.keywords=false; SELECT DISTINCT tweetmessage, user.name, createddate FROM tweets WHERE user.name = 'Aimee_Cottle'; A3 - STREAMING ARCHITECTURE HIVE/STORM 89 HIVE QUERY OF TWITTER DATA – QUERY DATA Failed with error messages
- 90. © Huxi LI, 2018 • Using another parser: • curl http://www.congiu.net/hive-json-serde/1.3.8/cdh5/json-serde-1.3.8-jar-with-dependencies.jar -O • curl http://www.congiu.net/hive-json-serde/1.3.8/cdh5/json-udf-1.3.8-jar-with-dependencies.jar -O • cp *.jar /usr/hdp/2.6.4.0-91/hive/lib/ & cp *.jar /usr/hdp/2.6.4.0-91/hive2/lib/ • Restart Hive (Ambari) • In Hive View2, execute: • ADD JAR /usr/hdp/2.6.4.0-91/hive2/lib/json-serde-1.3.8-jar-with-dependencies.jar; A3 - STREAMING ARCHITECTURE HIVE/STORM 90 HIVE QUERY OF TWITTER DATA – TRY ANOTHER PARSER
- 91. © Huxi LI, 2018 • Create tables ‘tweets2’ using OpenX: A3 - STREAMING ARCHITECTURE HIVE/STORM 91 HIVE QUERY OF TWITTER DATA – TRY ANOTHER PARSER CREATE EXTERNAL TABLE tweets2 (createddate string, geolocation string, tweetmessage string, `user` struct<geoenabled:boolean, id:int, name:string, screenname:string, userlocation:string>) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe’ LOCATION '/user/huxili/twitter_data/'
- 92. © Huxi LI, 2018 • Query twitter data on the new table: • SELECT * FROM tweets2 LIMIT 100; • It works ! A3 - STREAMING ARCHITECTURE HIVE/STORM 92 HIVE QUERY OF TWITTER DATA – TRY ANOTHER PARSER
- 93. © Huxi LI, 2018 • Query twitter data on the new table: • SET hive.support.sql11.reserved.keywords=false; SELECT tweetmessage, user.name, createddate FROM tweets2 WHERE user.screenname = 'Aimee_Cottle'; A3 - STREAMING ARCHITECTURE HIVE/STORM 93 HIVE QUERY OF TWITTER DATA – TRY ANOTHER PARSER OK
- 94. © Huxi LI, 2018 • Objective: • In this example, I will use STORM and twitter4j to analyzer real time tweets information, and then save the analysis result to HDFS of the Hadoop cluster. • Components: • TwitterSampleSpout – This is a STORM Spout responsible for reading of twitter stream using twitter4j (http://twitter4j.org). • WordSplitterBolt – This is a STORM Bolt responsible for splitting tweets into words for analysis. • IgnoreWordsBolt – This is a STORM Bolt responsible for filtering unwanted words. • WordCounterBolt – This is a STORM Bolt responsible for word frequency analysis. • HdfsWriterBolt – This is a STORM Bolt responsible for writing analysis result to HDFS. A3 - STREAMING ARCHITECTURE HIVE/STORM 94 STREAMING WITH STORM – TWITTER EXAMPLE
- 95. © Huxi LI, 2018 • Setup of twitter streaming (OAUTH): A3 - STREAMING ARCHITECTURE HIVE/STORM 95 STREAMING WITH STORM – TWITTER EXAMPLE
- 96. © Huxi LI, 2018 • STORM Topology: A3 - STREAMING ARCHITECTURE HIVE/STORM 96 STREAMING WITH STORM – TWITTER EXAMPLE StreamingSpout Tweets Splitter Bolt Filter Bolt Counter Bolt Aggregation & HDFS writing Bolt Hadoop HDFS
- 97. © Huxi LI, 2018 • Java Components: • AppContext: • Configuration of application. • HdfsWriterBolt: • Aggregation & Output to HDFS. • TwitterSampleSpout: • Streaming of tweets using twitter4j. • WordSplitterBolt, IgnoreWordsBolt, WordCounterBolt : • Splitting, filtering, and counting bolts. • Resources: • Configuration files for twitter4j and OAUTH authentication. A3 - STREAMING ARCHITECTURE HIVE/STORM 97 STREAMING WITH STORM – TWITTER EXAMPLE
- 98. © Huxi LI, 2018 • STORM Topology (Java): A3 - STREAMING ARCHITECTURE HIVE/STORM 98 STREAMING WITH STORM – TWITTER EXAMPLE
- 99. © Huxi LI, 2018 • Writing to HDSF (Java): A3 - STREAMING ARCHITECTURE HIVE/STORM 99 STREAMING WITH STORM – TWITTER EXAMPLE
- 100. © Huxi LI, 2018 • Packaging and execution: • Using maven • Plugin shade A3 - STREAMING ARCHITECTURE HIVE/STORM 100 STREAMING WITH STORM – TWITTER EXAMPLE
- 101. © Huxi LI, 2018 • Procedure of execution: 1. Create shaded jar (mvn clean install -Pcluster) 2. Uploading the generated application to HDFS using Ambari File View 3. Login onto slave1 (storm installed on slave1) 4. Perform the following operations: ‐ sudo su ‐ hdfs dfs -get /user/huxili/storm_apps/*.jar ‐ storm jar twitter-stream-example.jar com.huxili.storm.twitter.example.Topology 5. Observe the execution (see logs) 6. When the execution terminate, check the output file saved in HDFS (/user/huxili/storm_out/counts.txt) A3 - STREAMING ARCHITECTURE HIVE/STORM 101 STREAMING WITH STORM – TWITTER EXAMPLE
- 102. © Huxi LI, 2018 • Example execution logs: A3 - STREAMING ARCHITECTURE HIVE/STORM 102 STREAMING WITH STORM – TWITTER EXAMPLE
- 103. © Huxi LI, 2018 • Example execution logs: A3 - STREAMING ARCHITECTURE HIVE/STORM 103 STREAMING WITH STORM – TWITTER EXAMPLE
- 104. © Huxi LI, 2018 • Example results: A3 - STREAMING ARCHITECTURE HIVE/STORM 104 STREAMING WITH STORM – TWITTER EXAMPLE
- 105. © Huxi LI, 2018105 Progress. Together.