Pig on Spark
4 likes2,803 views
The document discusses transforming real-time data into decisions using technologies like Pig and Spark for ETL (Extract, Transform, Load) processes. It highlights features such as powerful in-memory processing, extendable libraries, and operator mappings. Additionally, it addresses issues related to scaling and performance, while inviting contributions and showcasing a cloud-agnostic approach.
Pig on Spark
- 1. “Transform Real Time Data into Real Time Decisions” Mayur Rustagi(@mayur_rustagi) “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 1
- 2. OPEN SOURCE CUSTOMERS PARTNERSHIPS “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 2
- 3. What steered us into Pig • DSL for ETL • Rich Operator Library • Extendable • Pluggable • Powerful ETL “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 3
- 4. What steered us into Spark • Powerful in-memory Processing • Simple operator on Data • Debuggable API • Efficient Execution • Universally distributed “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 4
- 5. Operator Mapping Pig Spark Load HadoopRDD Store saveasObjectFile Filter MappedRDD + filter func GroupBY (Local rearrange, global rearrange & package) Sort + Group by …. … “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 5
- 6. Current Flow “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 6
- 7. Filter Code implementation • https://bitbucket.org/SigmoidDev/spork/src/80a3e4626e4504 c1829568942e0690abc79d239a/src/org/apache/pig/backend/ hadoop/executionengine/spark/converter/FilterConverter.java ?at=spork-1.0 “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 7
- 8. Issues • Scaling • Performance • Spark Specific Operators (Cache) • Pig on Spark Unit test • Some specific joins & rank operation “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 8
- 9. Contribute • Pig on Spark Umbrella Jira • https://issues.apache.org/jira/browse/PIG-4059 • https://github.com/sigmoidanalytics/spork • Issues “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 9
- 10. What else is cool CloudFlux SigmaStream Cloud Deployment PIG/SQL Like DSL Fault Tolerance Rich Stream operators AutoScaling Multiple Data source/Sink Programmatic interface Add custom Operators Cloud Agnostic Apache Spark Based Apache License Apache License “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 10
- 11. Visit us @ P-15, Strata + Hadoop World “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 11
- 12. Thank You 1343 Kingfisher Way Sunnyvale, CA, 94087 India Office Gulmohar Enclave Road, Silver Spring Layout, Munnekollal Bengaluru, Karnataka 560037 US Office +1 (760) 203 3257 contact@sigmoidanalytics.com “Trans form Re a l Time Da t a into Re a l Time De c i s ions ” 12