Apache Falcon - Data Management Platform For Hadoop
Download as PPTX, PDF4 likes822 views
Apache Falcon is a data management platform for Hadoop that provides data processing, coordination of data pipelines, lifecycle management, and data discovery. It allows users to quickly configure data and processing tasks on Hadoop clusters. Falcon provides services for data relays, retention, replication, acquisition, and enables the holistic declaration of data pipelines and their dependencies. It also supports monitoring of data availability and processing through alerts and dashboards.
Apache Falcon - Data Management Platform For Hadoop
- 1. Apache Falcon Data Management Platform for Hadoop
- 4. ●Ajay Yadava ● Committer, Apache Falcon ● Lead - Apache Falcon @ Inmobi 4
- 5. What is Apache Falcon? Falcon is a data processing and management solution for Hadoop designed for data motion, coordination of data pipelines, lifecycle management, and data discovery. Falcon enables end consumers to quickly onboard their data and its associated processing and management tasks on Hadoop clusters. 5
- 6. 6
- 10. Data Retention as a service
- 12. Data Replication as a service
- 13. Data Acquisition As Service
- 14. 14
- 15. Holistic Declaration of Intent
- 17. Overview 17
- 18. Entity Dependency Graph Cluster Feed Process depends depends depends
- 19. Cluster specification 19 <cluster colo="SF-datacenter" description="" name="prod-cluster" xmlns="uri:falcon:cluster:0.1"> <interfaces> <interface type="readonly" endpoint="hftp://nn:50070" version="1.1.2"/> <interface type="write" endpoint="hdfs://nn:8020" version="1.1.2"/> <interface type="execute" endpoint="rm:8050" version="1.1.2"/> <interface type="workflow" endpoint="http://oozie:41000/oozie/" version="4.0.0"/> <interface type="registry" endpoint="http://oozie:41000/oozie/" version="4.0.0"/> <interface type="messaging" endpoint="tcp://:61616?daemon=true" version="5.4.3"/> </interfaces> <locations> <location name="staging" path="/projects/falcon/staging"/> <!--mandatory--> <location name="temp" path="/projects/falcon/tmp"/> <!--optional--> <location name="working" path="/projects/falcon/working"/> <!--optional--> </locations> </cluster> Used by distcp for replication Writing to HDFS Used to submit processes as MR Submit oozie jobs Used for alerts HDFS directories used by Falcon Hive metastore to register/deregister partitions & get data availability events
- 20. Feed specification <feed description="enhanced clicks replication feed" name="repl-feed" xmlns="uri:falcon:feed:0.1"> <frequency>minutes(5)</frequency> <late-arrival cut-off="hours(1)"/> <sla slaLow="hours(2)" slaHigh="hours(3)"/> <clusters> <cluster name="primary" type="source"> <validity start="2013-01-01T00:00Z" end="2030-01-01T00:00Z"/> <retention limit="days(2)" action="delete"/> </cluster> <cluster name="secondary" type="target"> <validity start="2013-11-15T00:00Z" end="2030-01-01T00:00Z"/> <retention limit="days(2)" action="delete"/> <locations> <location type="data" path="/data/clicks/repl-enhanced/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/> </locations> </cluster> </clusters> <locations> <location type="data" path="/data/clicks/enhanced/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/> </locations> <ACL owner="testuser-ut-user" group="group" permission="0x644"/> </feed> 20 Frequency Location SLA Monitoring Data Retention Data Replication
- 21. Process specification 21 <process name="clicks-hourly" xmlns="uri:falcon:process:0.1"> <clusters> <cluster name="corp"> <validity start="2011-11-02T00:00Z" end="2011-12-30T00:00Z"/> </cluster> <parallel>1</parallel> <order>LIFO</order> <frequency>hours(1)</frequency> <inputs> <input name="click" feed="clicks-enhanced" start="yesterday(0,0)" end="latest(0)" partition="*/US"/> </inputs> <outputs> <output name="clicksummary" feed="click-hourly" instance="today(0,0)"/> </outputs> <workflow name="test" version="1.0.0" engine="oozie" path="/user/guest/workflow" lib="/user/guest/workflowlib"/> <retry policy="periodic" delay="hours(10)" attempts="3"/> <late-process policy="exp-backoff" delay="hours(1)"> <late-input input="click" workflow-path="hdfs://clicks/late/workflow"/> </late-process> </process> Where should the process run? How should the process run? What to consume? What to produce? Late Data Handling Retry
- 22. Architecture 22
- 23. 23
- 24. 24
- 25. Falcon Unit A pipeline validation framework 25
- 26. Motivation for Falcon Unit ● User errors caught only at deploy time. ● Input/Output feeds and paths not getting resolved. ● Errors in specification. ● Integration Tests require environment setup/tearDown. ● Messy deployment scripts. ● Debugging was cumbersome. 26
- 27. Falcon Unit 27 Falcon Unit In Process execution env. ● Local Oozie ● Local File System ● Local Job Runner ● Local Message Queue Actual cluster ● Oozie ● HDFS ● YARN ● Active MQ Test suite
- 28. Example Process Submission: submit(EntityType.Process, <Path to Daily clicks Agg XML>); → Local submit(EntityType.Process, <Path to Daily clicks Agg XML>); → Cluster Mode Process Scheduling: scheduleProcess(“daily_clicks_agg”, startTime, numInstances, clusterName); Process Verification: getInstanceStatus(EntityType.Process,“daily_clicks_agg”, scheduleTime); 28
- 29. 29
- 30. 30
- 31. 31
- 32. 32
- 33. Deployment 33
- 34. 34 Embedded Mode
- 36. Monitoring 36
- 37. SLA Monitoring ●Alerts based on data Availability ●Dashboard ●Pluggable Alerting System ● Email ● JMS Notifications 37
- 38. Pipeline view 38
- 39. Triage 39
- 40. 40
- 41. ●Better Authentication and Authorization ●Even Better UI ●Even Better monitoring ●Process SLAs ●Streaming support ●A more powerful scheduler ●Pipeline Recovery 41
- 42. Community 42
- 43. 43
- 44. Questions? ●Apache Falcon ● falcon.apache.org ● dev@falcon.apache.org / user@falcon.apache.org ●Ajay Yadava ● ajayyadava@apache.org 44