Spark EMR AWS
- 1. Deploy a Spark Application on a Spark Cluster @ Elastic Map Reduce AWS Dr. Rim Moussa University of Carthage
- 2. Amazon S3 -Amazon Simple Storage Service Upload – S3 bucket for Spark code: .jar – S3 bucket for Data Uploads to S3 might be done via Terminal for big data sets laumch EC2 instance to upload data into S3 bucket curl ftp://ftp.ais.dk/ais_data/dk_csv_jun2018.zip | aws s3 cp - s3://aisdma Manipulation of S3 buckets and files can be done via Terminal aws s3 ls s3://data.info aws s3 cp s3://spark.jars/rm-1.0-veracity.jar . aws s3 rm s3://data.info 2
- 3. Open Datasets on Amazon Amazon has a repository of big datasets https://registry.opendata.aws/ Amazon implements a program AWS Public Dataset Program in order to democratize access to data and encourage the development of communities that benefit from access to shared datasets. https://aws.amazon.com/opendata/public-datasets/ 3
- 4. 4
- 5. 5
- 10. Hardware Tab 10
- 11. Cluster ready 11
- 12. ssh Master 12
- 13. → ssh Master 13
- 15. Click on EC2 instance of master 15
- 16. Review Master Security Groups 16
- 17. Check “allow pinging and ssh” 17
- 18. → ssh master 18
- 19. Steps towards submitting a workflow 19
- 20. Create S3 bucket for Workflow output 20
- 21. Submit job to Spark master 21
- 22. Job result 22
- 23. Re-check data.info S3 bucket 23