Automate all your EMR related activities
- 1. Automate all your EMR related activities Eitan Sela - System Architect eitan.sela@weissbeerger.com
- 2. $ whoami • "Hands-On" system Architect with more than 17 years of experience with billing, banking, information security (DLP) and Cloud IoT/Big Data applications. • Big Data specialist – Hadoop, Spark, Hive and EMR on AWS. • Work with vast AWS services, and with serverless projects especially. • Java development, scalability performance and stabilization expert. • Alexa skills developer. • Love to share my experience in lectures and meetups.
- 3. What to expect from this session • WeissBeerger use case – Aggregating raw orders and IoT data. • Amazon EMR basics. • Implementing ETLs with Spark. • Submitting work to a Cluster. • Provisioning scheduled transient EMR Clusters for ETLs jobs. • Our new Slack Chabot for EMR, using Amazon Lex!
- 4. WeissBeerger use case – Aggregating raw orders and IoT data
- 5. Solution • WeissBeerger bridges the gap between breweries, bars and customers.
- 6. Benefits for the brewery • Consumption Analytics. • Dynamic Promotions. • Beer Quality. • Value creation. • Beer penetration.
- 7. Benefits for the bar • Real Time Consumption Tracking. • Waste Reduction. • Sales Growth.
- 8. How does it work? • IoT (Pouring) – Beverage Analytics Hub. • Point of sales – POS Vendors, via REST API, S3, DB, etc.
- 9. Aggregating raw point of sales orders and IoT data
- 11. Amazon EMR - Easily Run and Scale Apache Hadoop, Spark, HBase, Presto, Hive, and other Big Data Frameworks
- 12. Amazon EMR – Create Cluster – Software and Steps
- 13. Amazon EMR – Create Cluster – Hardware
- 14. Amazon EMR – General Cluster Settings
- 15. Amazon EMR – Security
- 16. Implementing ETLs with Spark
- 17. Problem – Huge queries from MySQL to aggregative tables in Redshift
- 18. Solution - Implementing all ETL’s with PySpark
- 19. New data pipeline using EMR PySpark jobs – ELT rather than ETL
- 20. Submitting work to a Cluster
- 21. Launching Applications with spark-submit ./bin/spark-submit --jars jar1.jar,jar2.jar --py-files path/to/my/pymodule1.py, path/to/my/pymodule2.py my_program.py arg1 arg2 • The spark-submit script in Spark’s bin directory is used to launch applications on a cluster. • It can use all of Spark’s supported cluster managers through a uniform interface so you don’t have to configure your application especially for each one.
- 22. EMR Steps - Submit Work to a Cluster • You can submit work to a cluster by adding steps or by interactively submitting Hadoop jobs to the master node. • You can add steps to a cluster using the AWS Management Console, the AWS CLI, or the Amazon EMR API. • You can add step during cluster creation or to a running cluster.
- 23. EMR Steps – Job Types • Custom Jar. • Streaming Program. • Spark App. • Hive Program.
- 24. EMR Steps - Lifecycle • Pending • Cancelled (by user or API request) • Running • Completed / Failed.
- 25. EMR Steps – Add jobs to a running cluster
- 26. EMR Steps – View logs
- 27. WeissBeerger’s Spark ETL jobs submitted to EMR Cluster
- 28. Provisioning scheduled transient EMR Clusters for ETLs jobs
- 29. Requirements • Run ETL using Spark on EMR cluster every 1 hour for one month back. • Input: MySQL or Hive (stg). • Output: Hive (stg) or Redshift. • Storage should be separated from the compute, so EMR clusters should be transient. • Multiple clusters should be able to run together. • Fully automated and monitored.
- 31. Passing Spark Job steps parameters to Lambda input • We created a simple json with all parameters required to add step to EMR cluster.
- 32. Monitoring EMR Steps with Lambda and Datadog • We created a Lambda to sample all running EMR clusters for failed steps.
- 33. As more developers are developing PySpark Jobs…
- 34. Our new Slack Chabot for EMR, using Amazon Lex
- 35. Amazon Lex • Conversational interfaces for your applications. • Powered by the same deep learning technologies as Alexa. • Amazon Lex provides the advanced deep learning functionalities of automatic speech recognition (ASR) for converting speech to text, and natural language understanding (NLU).
- 36. Amazon Lex - Use cases - Call Center Bots
- 37. awsbot - the chatbot that help you manage AWS resources
- 38. awsbot - Demo
- 39. awsbot - Demo - EMR Cluster is ready
- 40. Q & A
- 41. We Are Hiring! Senior Data Scientist Senior Designer (UI/UX) Senior Full Stack Developer Java Developer Senior Manual QA Director of Ops BI Analyst Data Management Analyst Customer Success Manager Senior BI Analyst