Thailand Hadoop Big Data Challenge #1

Thailand Hadoop Big
Data Challenge #1
13-15 March 2015

2
Special thanks to Amazon Web Services
for supporting AWS's credit to run
EMR Hadoop cluster

3
Schedule
13 March
– 16.00 - 18.00 Workshop / Demo on Big Data Analytics
using Amazon EMR
– 18.00: Start registration for those who interested in running
the cluster for 30 Hours & Account access to Amazon EMR
will be given
14 March
– 06.00 Amazon EMR Cluster will be opened
– Participant will be discussed via online / Social Media
15 March (@ EGA Office)
– 12.00 Amazon EMR will be closed
– 13.00 Presentation by each competitor on the result
– 15.30 Winner Announcement

4
Architecture Overview of Amazon EMR

5
Hadoop Cluster for the challenge
10 AWS’s m3.xlarge EC2 server each with
4vCPU, 15 GByte Memory, 80 GB SSD Memory
A sample data set with more than 10 million
records will be given

6
Challenge rules
A competitor can use a sample data to analyse
with Hive, Pig or Map/Reduce
In addition, a competitor can use own large set of
data.
A winner will be judged from those who have a
best innovation / result from the analytics.
Those who are just would like to try using the
cluster are also welcome

7
Judging Criteria:
Complexity of the problem & Data Set 30%
Benefit to the society 20%
Innovation 30%
Presentation 20%

8
Judges
Assoc.Prof. Dr.Jirapun Daengdej
Mr. Danairat Thanabodithammachari
Dr.Thanachart Numnonda
Ms.Nantawan Wongkachonkitti

9
Awards
The best winner will receive an Apple TV.
Two winners will be selected for two free training
courses on
– Big Data using Hadoop Workshop; 30-31 March 2015
– Business Intelligence Design and Process; 18-20, 25-26
May 2015
Starbucks Card 200 Baht

10
EMR Cluster Setup
(This will be done by IMC Institute)

Thanachart Numnonda, thanachart@imcinstitute.com Feb 2015Big Data Hadoop on Amazon EMR – Hands On Workshop
Select EMR

Creating a cluster in EMR

Creating a cluster in EMR (cont.)
Name the cluster and also specify Log folder

Leave the Software Configuration as default

Leave the Hardware Configuration as default
Choose an exisitng EC2 key pair

Leave the others as default
Select Create Cluster

EMR Cluster Details
Note on the Master public DNS:
To see the details on how to connect to the Master Node using SSH click at SSH

19
Set Up an SSH Tunnel to the Master Node
– See instruction at
– http://docs.aws.amazon.com/ElasticMapReduce/latest/
DeveloperGuide/emr-ssh-tunnel.html

SSH Instruction for Mac/Linux

SSH Instruction for Windows

Connect to the master node

23
Launch the Hue Web Interface
Set Up an SSH Tunnel to the Master Node
– http://docs.aws.amazon.com/ElasticMapReduce/latest/Devel
operGuide/emr-ssh-tunnel.html
Configure Proxy Settings to View Websites
– http://docs.aws.amazon.com/ElasticMapReduce/latest/Devel
operGuide/emr-connect-master-node-proxy.html

24
Launch the Hue Web Interface (Cont.)
http://master-public-dns-name:8888/

Web Interface Host on EMR Cluster

28
Movielen Data
http://grouplens.org/datasets/movielens/
MovieLens 10M
(http://files.grouplens.org/datasets/movielens/ml-10m.zip)
– ratings.dat
– users.dat
– movies.dat

29
Transfer Data to Hadoop Cluster
wget http://files.grouplens.org/datasets/movielens/ml-10m.zip

31
Upload Data to Amazon S3
hadoop fs -put movies.csv s3://imcinstitute/data

34
Running Example
https://github.com/myui/hivemall/wiki/MovieLens-Dataset

Flight Details Data
http://stat-computing.org/dataexpo/2009/the-data.html

Data Description

Snapshot of Dataset

40
Registration
Provide your name, organization, mobile, e-mail
address
On-site registartion at 17.00 pm, 13 March
E-mail: contact@imcinstitute.com
Facebook message to Thanachart Numnonda
Your username & password & key & public DNS will
be send to your e-mail by 6 am, 14 March

41
On-line communication
Facebook Group: Hadoop-Thailand
Line group
Facebook message
E-mail to contact@imcinstitute.com

42
www.facebook.com/imcinstitute

43
Thank you
thanachart@imcinstitute.com
www.facebook.com/imcinstitute
www.slideshare.net/imcinstitute
www.thanachart.org

Thailand Hadoop Big Data Challenge #1

More Related Content

What's hot (20)

Viewers also liked (19)

Similar to Thailand Hadoop Big Data Challenge #1 (20)

More from IMC Institute (20)

Recently uploaded (20)

Thailand Hadoop Big Data Challenge #1