2014 Import.io Data Summit - Including Hadoop/Impala Getting Started Demo
1 like514 views
This document discusses large-scale data analysis using AWS, highlighting the importance of generating, storing, analyzing, and sharing data. It covers various AWS services such as Amazon S3, Amazon EC2, and Amazon Elastic MapReduce, demonstrating data management and real-time processing capabilities. The content emphasizes the cost-effectiveness and scalability of AWS for managing vast amounts of data across different industries.
2014 Import.io Data Summit - Including Hadoop/Impala Getting Started Demo
- 1. ianmas@amazon.com @IanMmmm LARGE SCALE DATA ANALYSIS WITH AWS Ian Massingham – Technical Evangelist
- 2. THE MORE DATA YOU COLLECT THE MORE VALUE YOU CAN DERIVE FROM IT!
- 5. THE COST OF DATA GENERATION IS FALLING!
- 6. We are constantly producing more data
- 7. From all types of industries
- 10. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
- 11. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE! Lower cost, higher throughput
- 12. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE! Lower cost, higher throughput Highly constrained
- 13. + ELASTIC AND HIGHLY SCALABLE + NO UPFRONT CAPITAL EXPENSE + ONLY PAY FOR WHAT YOU USE + AVAILABLE ON-DEMAND = REMOVE CONSTRAINTS
- 14. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
- 15. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE! AWS Import / Export AWS Direct Connect
- 16. Inbound data transfer is free Multipart upload to S3 Physical media AWS Direct Connect
- 17. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE! Amazon S3, Amazon Glacier, Amazon DynamoDB, Amazon RDS, Amazon Redshift, AWS Storage Gateway, Data on Amazon EC2
- 18. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE! Amazon EC2 Amazon Elastic MapReduce
- 20. AMAZON ELASTIC MAPREDUCE HADOOP AS A SERVICE!
- 21. • SPLITS DATA INTO PIECES • LETS PROCESSING OCCUR • GATHERS THE RESULTS!
- 22. HDFS EMRKinesis S3 DynamoDB Data management Pig Analytics languages/engines RDS Redshift AWS Data Pipeline
- 23. EMR + IMPALA DEMO
- 24. STARTING AN EMR CLUSTER WITH HADOOP ECOSYSTEM TOOLS PRE-INSTALLED
- 25. COPY & LOAD OUR DATASET $ scp –i EMRKeyPair.pem ~/aws/hadoop/LHRarrivals*.csv hadoop@ec2-‐54-‐76-‐242-‐238.eu-‐ west-‐1.compute.amazonaws.com: $ ssh –i EMRKeyPair.pem hadoop@ec2-‐54-‐76-‐242-‐238.eu-‐west-‐1.compute.amazonaws.com $ hadoop fs -‐mkdir /data/ $ hadoop fs -‐put <uploaded_files> /data/ $ hadoop fs -‐ls -‐h -‐R /data/ or at scale, Distributed Copy using S3DistCp to parallel load from S3 $ . /home/hadoop/impala/conf/impala.conf $ hadoop jar /home/hadoop/lib/emr-‐s3distcp-‐1.0.jar -‐Dmapreduce.job.reduces=30 -‐-‐ src s3://s3bucketname/ -‐-‐dest hdfs://$HADOOP_NAMENODE_HOST:$HADOOP_NAMENODE_PORT/ data/ -‐-‐outputCodec 'none' ** Run on a cluster master node
- 26. CREATE EXTERNAL TABLE $ #check the size of our data set $ wc –l LHRarrivals*.csv 850 LHRarrivals2.csv 1526 LHRarrivals.csv 2376 total $ impala-‐shell Welcome to the Impala shell. > create EXTERNAL TABLE flights ( input STRING, id BIGINT, widget STRING, source STRING, resultnum BIGINT, pageurl STRING, scheduled STRING, flightnumber STRING, airport STRING, status STRING, terminal STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/data/'; > select count (*) from flights; Should return count(*) 2376 reflecting the size of the data set
- 27. DEMO OF ODBC ACCESS Doing this part on Amazon WorkSpaces using the Simba Cloudera Impala ODBC Driver.! Set up an SSH tunnel to the master node to allow us to connect to port 25010 from the WorkSpaces desktop to the Impala ODBC port! A previously configured system DSN allows us to work with the data from our EMR/Impala cluster directly within Microsoft Excel!
- 28. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE! Amazon S3, Amazon DynamoDB, Amazon RDS, Amazon Redshift, Data on Amazon EC2
- 29. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
- 30. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE! BATCH PROCESSING
- 31. GENERATE ➔ ➔ SHARE! STREAM PROCESSING
- 32. AMAZON KINESIS REAL-TIME DATA STREAM PROCESSING!
- 33. Real-time response to content in semi-structured data streams Relatively simple computations on data (aggregates, filters, sliding window, etc.)
- 34. Hourly server logs: how your systems went wrong an hour ago Weekly / Monthly Bill: What you spent this past billing cycle Daily customer report from your website: tells you what deal or ad to try next time Daily fraud reports: tells you if there was fraud yesterday Daily business reports: tells me how customers used AWS services yesterday Real-time metrics: what just went wrong now Real-time spending alerts/caps: guaranteeing you can’t overspend Real-time analysis: what to offer the current customer now Real-time detection: blocks fraudulent use now Fast ETL into Amazon Redshift: how are customers using services now
- 35. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
- 36. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE! Amazon S3, Amazon DynamoDB, Amazon RDS, Amazon Redshift, Data on Amazon EC2 Amazon EC2 Amazon Elastic MapReduce Amazon S3, Amazon Glacier, Amazon DynamoDB, Amazon RDS, Amazon Redshift, AWS Storage Gateway, Data on Amazon EC2 AWS Import / Export AWS Direct Connect
- 37. GENERATE ➔ ➔ SHARE! STREAM PROCESSING
- 38. GENERATE ➔ ➔ SHARE! STREAM PROCESSING Amazon S3, Amazon DynamoDB, Amazon RDS, Amazon Redshift, Data on Amazon EC2 Amazon Kinesis Stream Processing on Amazon EC2
- 39. WANT TO KNOW MORE? aws.amazon.com/solutions/case-studies/big-data/!
- 40. ianmas@amazon.com @IanMmmm LARGE SCALE DATA ANALYSIS WITH AWS Ian Massingham – Technical Evangelist